Book Image

Pentaho Data Integration Beginner's Guide - Second Edition - Second Edition

By : María Carina Roldán
Book Image

Pentaho Data Integration Beginner's Guide - Second Edition - Second Edition

By: María Carina Roldán

Overview of this book

Capturing, manipulating, cleansing, transferring, and loading data effectively are the prime requirements in every IT organization. Achieving these tasks require people devoted to developing extensive software programs, or investing in ETL or data integration tools that can simplify this work. Pentaho Data Integration is a full-featured open source ETL solution that allows you to meet these requirements. Pentaho Data Integration has an intuitive, graphical, drag-and-drop design environment and its ETL capabilities are powerful. However, getting started with Pentaho Data Integration can be difficult or confusing. "Pentaho Data Integration Beginner's Guide - Second Edition" provides the guidance needed to overcome that difficulty, covering all the possible key features of Pentaho Data Integration. "Pentaho Data Integration Beginner's Guide - Second Edition" starts with the installation of Pentaho Data Integration software and then moves on to cover all the key Pentaho Data Integration concepts. Each chapter introduces new features, allowing you to gradually get involved with the tool. First, you will learn to do all kinds of data manipulation and work with plain files. Then, the book gives you a primer on databases and teaches you how to work with databases inside Pentaho Data Integration. Moreover, you will be introduced to data warehouse concepts and you will learn how to load data in a data warehouse. After that, you will learn to implement simple and complex processes. Finally, you will have the opportunity of applying and reinforcing all the learned concepts through the implementation of a simple datamart. With "Pentaho Data Integration Beginner's Guide - Second Edition", you will learn everything you need to know in order to meet your data manipulation requirements.
Table of Contents (26 chapters)
Pentaho Data Integration Beginner's Guide
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Best Practices
Index

Solutions to commonly occurring situations


Among several common use-cases in PDI, there are three that have been specially addressed in PDI 5: restartability, database transactions, and looping.

Restartability has to do with the ability of restarting a job after an interruption. To keep track of the status of an execution, you define checkpoints. Checkpoints ensure that the status variable, arguments, files, and rows from the result set are serialized. This way, the execution of a failed job can be easily resumed in a safe state after the following job entry of the last successful checkpoint.

The second common requirement that was introduced is the database transaction across transformations and jobs. This differs from previous versions of PDI where the scope of a transaction was a single transformation or job.

Finally, a very interesting feature included in PDI 5 is the possibility of including subjobs in transformations, through the use of executors. For a long time PDI developers used to ask, "Can I run a job inside a transformation?". The answer was definitely a no. In order to solve the requirement, the solution was to create jobs and transformations nested in complex ways. Now you can avoid all that unnecessary work by looping-over data or files in an easier way. There is a job executor step that can easily be configured to loop-over the rows in a dataset. Not only is the loop easier to implement, but also there is a bonus: the step returns the execution results (number of rows read, number of errors, and so on), the result rows, and the result files. Analogous to this, there is also a transformation executor.