Talend Open Studio Cookbook

Talend Open Studio Cookbook

By : Rick Barton

Buy this Book

Talend Open Studio Cookbook

By: Rick Barton

Buy this Book

Overview of this book

Data integration is a key component of an organization's technical strategy, yet historically the tools have been very expensive. Talend Open Studio is the world's leading open source data integration product and has played a huge part in making open source data integration a popular choice for businesses worldwide.This book is a welcome addition to the small but growing library of Talend Open Studio resources. From working with schemas to creating and validating test data, to scheduling your Talend code, you will get acquainted with the various Talend database handling techniques. Each recipe is designed to provide the key learning point in a short, simple and effective manner.This comprehensive guide provides practical exercises that cover all areas of the Talend development lifecycle including development, testing, debugging and deployment. The book delivers design patterns, hints, tips, and advice in a series of short and focused exercises that can be approached as a reference for more seasoned developers or as a series of useful learning tutorials for the beginner.The book covers the basics in terms of schema usage and mappings, along with dedicated sections that will allow you to get more from tMap, files, databases and XML. Geared towards the whole lifecycle, the Talend Open Studio Cookbook shows readers great ways to handle everyday tasks, and provides an insight into all areas of a development cycle including coding, testing, and debugging of code to provide start-to-finish coverage of the product.

Talend Open Studio Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Preface

Free Chapter

Introduction and General Principles

Before you begin

Installing the software

Enabling tHashInput and tHashOutput

Metadata and Schemas

Introduction

Hand-cranking a built-in schema

Propagating schema changes

Creating a generic schema from the existing metadata

Cutting and pasting schema information

Dropping schemas to empty components

Creating schemas from lists

Validating Data

Introduction

Enabling and disabling reject flows

Gathering all rejects prior to killing a job

Validating against the schema

Rejecting rows using tMap

Checking a column against a list of allowed values

Checking a column against a lookup

Creating validation rules for more complex requirements

Creating binary error codes to store multiple test results

Mapping Data

Introduction

Simple mapping and tMap time savers

Creating tMap expressions

Using the ternary operator for conditional logic

Using intermediate variables in tMap

Filtering input rows

Splitting an input row into multiple outputs based on input conditions

Joining data using tMap

Hierarchical joins using tMap

Using reload at each row to process real-time / near real-time data

Using Java in Talend

Introduction

Performing one-off pieces of logic using tJava

Setting the context and globalMap variables using tJava

Adding complex logic into a flow using tJavaRow

Creating pseudo components using tJavaFlex

Creating custom functions using code routines

Importing JAR files to allow use of external Java classes

Managing Context Variables

Introduction

Creating a context group

Adding a context group to your job

Adding contexts to a context group

Using tContextLoad to load contexts

Using implicit context loading to load contexts

Turning implicit context loading on and off in a job

Setting the context file location in the operating system

Working with Databases

Introduction

Setting up a database connection

Importing the table schemas

Reading from database tables

Using context and globalMap variables in SQL queries

Printing your input query

Writing to a database table

Printing your output query

Managing database sessions

Passing a session to a child job

Selecting different fields and keys for insert, update, and delete

Capturing individual rejects and errors

Database and table management

Managing surrogate keys for parent and child tables

Rewritable lookups using an in-process database

Managing Files

Introduction

Appending records to a file

Reading rows using a regular expression

Using temporary files

Storing intermediate data in the memory using tHashMap

Reading headers and trailers using tMap

Reading headers and trailers with no identifiers

Using the information in the header and trailer

Adding a header and trailer to a file

Moving, copying, renaming, and deleting files and folders

Capturing file information

Processing multiple files at once

Processing control/validation files

Creating and writing files depending on the input data

Working with XML, Queues, and Web Services

Introduction

Using tXMLMap to read XML

Using tXMLMap to create an XML document

Reading complex hierarchical XML

Writing complex XML

Calling a SOAP web service

Calling a RESTful web service

Reading and writing to a queue

Ensuring lossless queues using sessions

Debugging, Logging, and Testing

Introduction

Find the location of compilation errors using the Problems tab

Locating execution errors from the console output

Using the Talend debug mode – row-by-row execution

Using the Java debugger to debug Talend jobs

Using tLogRow to show data in a row

Using tJavaRow to display row information

Using tJava to display status messages and variables

Printing out the context

Dumping the console output to a file from within a job

Creating simple test data using tRowGenerator

Creating complex test data using tRowGenerator, tFlowToIterate, tMap, and sequences

Creating random test data using lookups

Creating test data using Excel

Testing logic – the most-used pattern

Killing a job from within tJavaRow

Deploying and Scheduling Talend Code

Introduction

Creating compiled executables

Using a different context

Adding command-line context parameters

Managing job dependencies

Capturing and acting on different return codes

Returning codes from a child job without tDie

Passing parameters to a child job

Executing non-Talend objects and operating system commands

Common Mistakes and Other Useful Hints and Tips

Introduction

My tab is missing

Finding the code routine

Finding a new context variable

Reloads going missing at each row global variable

Dragging component globalMap variables

Some complex date formats

Capturing tMap rejects

Adding job name, project name, and other job specific information

Printing tMap variables

Stopping memory errors in Talend

Common Type Conversions

Management of Contexts

Introduction

Manipulating contexts in Talend Open Studio

Understanding implicit context loading

Understanding tContextLoad

Manually checking and setting contexts

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Before you begin

Before you begin the exercises in the book, it is worth becoming familiar with some of the key concepts and best practices.

Keep code changes small and test often

When developing using Talend, as with any other development tool, it is recommended to code in short bursts and test (run) frequently.

By keeping each change small, it is much easier to find where and what has caused problems during compilation and execution.

Chapter 10, Debugging, Logging, and Testing, is dedicated to debugging and logging; however, observing the preceding method will save time having to perform debugging steps that can sometimes take a long time.

Document your code

Talend sub-jobs have the ability to add titles, and every component in Talend has the option to add documentation for the component. Where you use Java, you should use the Java comment structures to document the code. Remember to use all these methods as you go along to ensure that your code is well documented.

Contexts and globalMap

context and globalMap are global areas used to store data that can be used by all components within a Talend job.

context variables are predefined prior to job execution in a context group, whereas globalMap variables are created on the fly at any point within a job.

Context variables

Context variables are used by Talend to store parameter information, and can be used:

To pass information into a job from the command line and/or a parent job
To manage values of parameters between environments
To store values within a job or set of jobs

Chapter 6, Managing Context Variables, is dedicated to the use and management of context variables within Talend

globalMap

globalMap is a very important construct within Talend, in that:

Almost every component will write information to globalMap once it completes execution (for example NB_LINE is the number of rows processed in a component).
Certain components, such as tFlowToIterate or tFileList, will store data in globalMap variables for use by downstream components.
Developers can read and write to globalMap to create global variables in an ad hoc fashion. The use of global variables can often be the best way to ensure code is simple and efficient.

Java

Talend is a Java code generator, so having a little Java knowledge can help when using Talend. There are many Java tutorials for beginners online, and a little time spent learning the basics will help speed up your understanding of Talend.

Other background knowledge

As a data integrator, you will be expected to understand many technologies and how to interface with them, and this book assumes a basic knowledge of many of the most frequent data sources and targets.

Chapter 7, Working with Databases, relates to using Talend with databases. We have chosen to use MySQL, because it is quick to install, simple to use, and readily available. Basic knowledge of SQL and MySQL will therefore be required to perform the exercises in this chapter.

Other chapters will also assume knowledge of csv files, MS Excel, XML, and web services.

Talend Open Studio Cookbook

By : Rick Barton

Talend Open Studio Cookbook

By: Rick Barton

Overview of this book

Related Content you might be interested in

Current Title:

Talend Open Studio Cookbook

Before you begin