Book Image

Talend Open Studio Cookbook

By : Rick Barton
Book Image

Talend Open Studio Cookbook

By: Rick Barton

Overview of this book

Data integration is a key component of an organization's technical strategy, yet historically the tools have been very expensive. Talend Open Studio is the world's leading open source data integration product and has played a huge part in making open source data integration a popular choice for businesses worldwide.This book is a welcome addition to the small but growing library of Talend Open Studio resources. From working with schemas to creating and validating test data, to scheduling your Talend code, you will get acquainted with the various Talend database handling techniques. Each recipe is designed to provide the key learning point in a short, simple and effective manner.This comprehensive guide provides practical exercises that cover all areas of the Talend development lifecycle including development, testing, debugging and deployment. The book delivers design patterns, hints, tips, and advice in a series of short and focused exercises that can be approached as a reference for more seasoned developers or as a series of useful learning tutorials for the beginner.The book covers the basics in terms of schema usage and mappings, along with dedicated sections that will allow you to get more from tMap, files, databases and XML. Geared towards the whole lifecycle, the Talend Open Studio Cookbook shows readers great ways to handle everyday tasks, and provides an insight into all areas of a development cycle including coding, testing, and debugging of code to provide start-to-finish coverage of the product.

Related Content you might be interested in

Current Title:

Small book image
Talend Open Studio Cookbook
Rick Barton
Oct, 2013 | 270 pages
No titles found
Table of Contents (21 chapters)
Talend Open Studio Cookbook
Talend Open Studio Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Introduction and General Principles
Introduction and General Principles
Before you begin
Installing the software
Enabling tHashInput and tHashOutput
2
Metadata and Schemas
Metadata and Schemas
Introduction
Hand-cranking a built-in schema
Propagating schema changes
Creating a generic schema from the existing metadata
Cutting and pasting schema information
Dropping schemas to empty components
Creating schemas from lists
3
Validating Data
Validating Data
Introduction
Enabling and disabling reject flows
Gathering all rejects prior to killing a job
Validating against the schema
Rejecting rows using tMap
Checking a column against a list of allowed values
Checking a column against a lookup
Creating validation rules for more complex requirements
Creating binary error codes to store multiple test results
4
Mapping Data
Mapping Data
Introduction
Simple mapping and tMap time savers
Creating tMap expressions
Using the ternary operator for conditional logic
Using intermediate variables in tMap
Filtering input rows
Splitting an input row into multiple outputs based on input conditions
Joining data using tMap
Hierarchical joins using tMap
Using reload at each row to process real-time / near real-time data
5
Using Java in Talend
Using Java in Talend
Introduction
Performing one-off pieces of logic using tJava
Setting the context and globalMap variables using tJava
Adding complex logic into a flow using tJavaRow
Creating pseudo components using tJavaFlex
Creating custom functions using code routines
Importing JAR files to allow use of external Java classes
6
Managing Context Variables
Managing Context Variables
Introduction
Creating a context group
Adding a context group to your job
Adding contexts to a context group
Using tContextLoad to load contexts
Using implicit context loading to load contexts
Turning implicit context loading on and off in a job
Setting the context file location in the operating system
7
Working with Databases
Working with Databases
Introduction
Setting up a database connection
Importing the table schemas
Reading from database tables
Using context and globalMap variables in SQL queries
Printing your input query
Writing to a database table
Printing your output query
Managing database sessions
Passing a session to a child job
Selecting different fields and keys for insert, update, and delete
Capturing individual rejects and errors
Database and table management
Managing surrogate keys for parent and child tables
Rewritable lookups using an in-process database
8
Managing Files
Managing Files
Introduction
Appending records to a file
Reading rows using a regular expression
Using temporary files
Storing intermediate data in the memory using tHashMap
Reading headers and trailers using tMap
Reading headers and trailers with no identifiers
Using the information in the header and trailer
Adding a header and trailer to a file
Moving, copying, renaming, and deleting files and folders
Capturing file information
Processing multiple files at once
Processing control/validation files
Creating and writing files depending on the input data
9
Working with XML, Queues, and Web Services
Working with XML, Queues, and Web Services
Introduction
Using tXMLMap to read XML
Using tXMLMap to create an XML document
Reading complex hierarchical XML
Writing complex XML
Calling a SOAP web service
Calling a RESTful web service
Reading and writing to a queue
Ensuring lossless queues using sessions
10
Debugging, Logging, and Testing
Debugging, Logging, and Testing
Introduction
Find the location of compilation errors using the Problems tab
Locating execution errors from the console output
Using the Talend debug mode – row-by-row execution
Using the Java debugger to debug Talend jobs
Using tLogRow to show data in a row
Using tJavaRow to display row information
Using tJava to display status messages and variables
Printing out the context
Dumping the console output to a file from within a job
Creating simple test data using tRowGenerator
Creating complex test data using tRowGenerator, tFlowToIterate, tMap, and sequences
Creating random test data using lookups
Creating test data using Excel
Testing logic – the most-used pattern
Killing a job from within tJavaRow
11
Deploying and Scheduling Talend Code
Deploying and Scheduling Talend Code
Introduction
Creating compiled executables
Using a different context
Adding command-line context parameters
Managing job dependencies
Capturing and acting on different return codes
Returning codes from a child job without tDie
Passing parameters to a child job
Executing non-Talend objects and operating system commands
12
Common Mistakes and Other Useful Hints and Tips
Common Mistakes and Other Useful Hints and Tips
Introduction
My tab is missing
Finding the code routine
Finding a new context variable
Reloads going missing at each row global variable
Dragging component globalMap variables
Some complex date formats
Capturing tMap rejects
Adding job name, project name, and other job specific information
Printing tMap variables
Stopping memory errors in Talend
Common Type Conversions
Common Type Conversions
Management of Contexts
Management of Contexts
Introduction
Manipulating contexts in Talend Open Studio
Understanding implicit context loading
Understanding tContextLoad
Manually checking and setting contexts
Index
Index

Reading headers and trailers using tMap

This recipe shows how to parse a file that has header and trailer records, and a record type at the start of a line.

Getting ready

Open the jo_cook_ch08_0060_headTrailtMap job.

How to do it...

The steps for reading headers and trailers using tMap are as follows:

  1. Drag a tMap component onto the canvas.

  2. Connect the tFileInputFullRow to tMap, and rename the flow to customerIn.

  3. Open tMap, and create three new outputs. Name them header, detail, and trailer.

  4. Copy the input field line into each of the new outputs.

  5. Add the expression filter customerIn.line.startsWith("00") to the header output table.

  6. Add the expression filter customerIn.line.startsWith("01") to the detail output table.

  7. Add the expression filter customerIn.line.startsWith("99") to the trailer output table.

  8. Your tMap should now look like the one shown as follows:

  9. Close tMap, and drag three tExtractDelimitedFields components to the canvas, along with three tLogRow components.

  10. Join each output from tMap to each...

Previous Section
End of Section 7
Next Section