The history of Kettle began in 2001 when Matt Caster, Pentaho Data Integration's chief architect and creator of Kettle, was working as a BI consultant. He had the idea of writing his own ETL tool to have a better and cheaper way to transfer data from one place to another. He was looking for a different solution, something that was better than inventing ugly data warehouse solutions written in PL/SQL, VB, or Shell scripts. He spent two years doing a thorough analysis of the problem. Because he was busy all the time with his work as a consultant, he worked on this project either during the weekends or at night. After this phase, he came out with a set of analyses documents and a couple of test programs written in C. He was not fully satisfied with what he got, so by early 2003, he started looking towards Java and continued his work on the product on this platform that, in those years, was gaining more traction in the market. So by the mid of 2003, the first version of the ETL design tool named Stir (which is now called Spoon) came to life.

It is interesting to see a screenshot of how things were then:

Stir featured a big X on the graphical view, and the log view was not working and neither were most step dialogs; but, it is useful for you to understand what the starting point of this adventure was. A certain number of other releases came out, each with a different set of new features or bugs fixed.

In 2004, work was reasonably stable and he was able to deploy Kettle for the first time to a customer. Because of the "real-world" situation, a lot of things needed to be fixed and new features needed to be implemented. That was why, in those days, things were advancing a lot faster than they were in the first three years. It seemed that the code base grew so fast that several refactorings and code cleanings were needed. Version 2.0 was one of the last "unstructured" versions. But it was thanks to the Java expertise from companies such as ixor (Wim De Clerq especially) that Kettle survived and changed radically. They helped Matt a lot with refactoring and code reorganizations to give the application a better structure and to simplify the code. At that time, Kettle had a fairly complete first release with support for slow-changing dimensions, junk dimensions, 28 steps, and 13 database connectors.

The application that was initially closed source was open sourced in late 2005. The first version under this new licensing mode was published in December 2005, and the response from the community was massive.

As of today, PDI is one of the best ETL open source solutions; it is made up of the following components:

Designing a simple PDI transformation (Simple) shows you how to design the simple transformation used as an example throughout all the recipes in this book. It also summarizes how we can develop a simple transformation using the design tool Spoon and some advises to follow in the development of transformations.

Designing a simple PDI job (Simple) shows you how to design a simple job that uses the transformation developed in the previous recipe. This job will be used as an example throughout all the book recipes. Like the previous recipe, it helps summarize how we can develop a simple job using the design tool Spoon and some advises to follow in the development of jobs and transformations.

Configuring command-line tools to run properly (Simple) represents the main starting point for everything. You will find what the main things are that you need to do to configure your PDI ETL system properly so that anything is able to work without any inconvenience.

Executing PDI jobs from a filesystem (Simple) is the first of a set of three recipes about how to start an ETL job from the command line. This is about how to start your PDI process when it is saved to the regular filesystem.

Executing PDI jobs packaged in archive files (Intermediate) explains the same topic as the previous recipe, but considers the process files to be packaged as an archive file. This is useful any time you use an ETL procedure on multiple systems (I mean for examples where you want to do some maintenance procedure) and you want to quickly move and run it without pain.

Executing PDI jobs from the repository (Simple) is the last in the series about how to start a job or transformation from the command line. This recipe is all about starting a job or transformation were the ETL files are stored in the repository.

Dealing with the execution log (Simple) explains how to efficiently use the various types of arguments available to manage the logfile and how to set the appropriate severity depending on the situation.

Discovering your PDI repository from the command line (Simple) is useful any time you decide to explore your PDI repository from the command line. It could so happen that you may forget what you have in your repository and where you have placed it. If that is the case, this is the recipe for you.

Exporting jobs and transformations to .zip files (Simple) shows you how to use a very simple and useful export mechanism. It could be useful to create a backup of your process files or to export them and easily move them to other systems.

Managing return code of PDI processes (Simple) is really the recipe for you if you need to get the procedure's return code to manage the conditional execution of other external processes.

Scheduling PDI jobs and transformations (Intermediate) tries to clear any doubts you have about scheduling your ETL processes.

To run the samples in this book, you need a version of Java installed (JDK 1.6 or higher is fine) and you need the latest version of Pentaho Data Integration. If you don't have PDI installed, you can freely download it from this link: http://kettle.pentaho.com. For those of you who prefer to compile the version directly from the sources (I prefer to do this for my personal installation), you can get the latest sources from the following repository link: svn://source.pentaho.org/svnkettleroot.

This book is for ETL developers with any amount of knowledge of PDI, from basic to advanced, and who already have knowledge about developing ETL processes using PDI. It is a book for anyone who wants to get a better idea about how to get their ETL processes running anywhere, manually or by scheduling, using command-line tools. You will have all the knowledge needed to do your work easily and without pain.

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text are shown as follows: "We can include other contexts through the use of the include directive".

A block of code is set as follows:

if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS=-Xmx512m

Any command-line input or output is written as follows:

$ kitchen.sh –file:/home/sramazzina/tmp/samples/export-job.kjb

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "Click on the New button from the toolbar menu".

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.