Pentaho Data Integration (also known as Kettle) is one of the leading open source data integration solutions. With Kettle, you can take data from a multitude of sources, transform and conform the data to given requirements, and load the data into just as many target systems. Not only is PDI capable of transforming and cleaning data, it also provides an ever-growing number of plugins to augment what is already a very robust list of features.
Pentaho Data Integration Cookbook Second Edition picks up where the first edition left off, by updating the recipes to the latest edition of PDI and diving into new topics working with Big Data and cloud sources, data analytics, and more.
Pentaho Data Integration Cookbook Second Edition shows you how to take advantage of all the aspects of Kettle through a set of practical recipes organized to find quick solutions to your needs. The book starts with showing you how to work with data sources files, relational databases, . Then we go into how to work with data streams merging data from different sources, how to take advantage of the different tools to clean up and transform data, and how to build nested jobs and transformations. More advanced topics are also covered, data analytics, data visualization, plugins, and integration of Kettle with other tools in the Pentaho suite.
Pentaho Data Integration Cookbook Second Edition provides recipes with easy step-by-step instructions to accomplish specific tasks. The code for the recipes can be adapted and built upon to meet individual needs.
Chapter 1, Working with Databases, shows you how to work with relational databases with Kettle. The recipes show you how to create and share database connections, perform typical database functions (select, insert, update,delete), as well as more advanced tricks building and executing queries at runtime.
Chapter 2, Reading and Writing Files, not only shows you how to read and write files, but also how to work with semi-structured files, and read data from Amazon Web Services.
Chapter 3, Working with Big Data and Cloud Sources, covers how to load and read data from some of the many different NoSQL data sources as well as from Salesforce.com.
Chapter 4, Manipulating XML Structures, shows you how to read, write, and validate XML. Simple and complex XML structures are shown as well as more specialized formats RSS feeds.
Chapter 5, File Management, demonstrates how to copy, move, transfer, and encrypt files and directories.
Chapter 6, Looking for Data, shows you how to search for information through various methods via databases, web services, files, and more. This chapter also shows you how to validate data with Kettle's builtin validation steps.
Chapter 7, Understanding and Optimizing Data Flows, details how Kettle moves data through jobs and transformations and how to optimize data flows.
Chapter 8, Executing and Re-using Jobs and Transformations, shows you how to launch jobs and transformations in various ways through static or dynamic arguments and parameterization. Objectoriented transformations through subtransformations also explained.
Chapter 9, Integrating Kettle and the Pentaho Suite, works with some of the other tools in the Pentaho suite to show how combining tools provides even more capabilities and functionality for reporting, dashboards, and more.
Chapter 10, Getting the Most Out of Kettle, works with some of the commonly needed features (e-mail and logging) as well as building sample data sets, and using Kettle to read meta information on jobs and transformations via files or Kettle's database repository.
Chapter 11, Utilizing Visualization Tools in Kettle, explains how to work with plugins and focuses on DataCleaner, AgileBI, and Instaview, an Enterprise feature that allows for fast analysis of data sources.
Chapter 12, Data Analytics, shows you how to work with the various analytical tools built into Kettle, focusing on statistics gathering steps and building datasets for Weka.
Appendix A, Data Structures, shows the different data structures used throughout the book.
Appendix B, References, provides a list of books and other resources that will help you connect with the rest of the Pentaho community and learn more about Kettle and the other tools that are part of the Pentaho suite.
PDI is written in Java. Any operating system that can run JVM 1.5 or higher should be able to run PDI. Some of the recipes will require other software, as listed:
Hortonworks Sandbox: This is Hadoop in a box, a great environment to learn how to work with NoSQL solutions without having to install everything.
Web Server with ASP support: This is needed for two recipes to show how to work with web services.
DataCleaner: This is one of the top open source data profiling tools and integrates with Kettle.
MySQL: All the relational database recipes have scripts for MySQL provided. Feel free to use another relational database for those recipes.
In addition, it's recommended to have access to Excel or Calc and a decent text editor (like Notepad++ or gedit).
Having access to an Internet connection will be useful for some of the recipes that use cloud services, as well as making it possible to access the additional links that provide more information about given topics throughout the book.
If you are a software developer, data scientist, or anyone else looking for a tool that will help extract, transform, and load data as well as provide the tools to perform analytics and data cleansing, then this book is for you! This book does not cover the basics of PDI, SQL, database theory, data profiling, and data analytics.
In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Copy the .jar file containing the driver to the lib directory inside the Kettle installation directory."
A block of code is set as follows:
"lastname","firstname","country","birthyear" "Larsson","Stieg","Swedish",1954 "King","Stephen","American",1947 "Hiaasen","Carl ","American",1953
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
<request>
<type>City</type>
<query>Buenos Aires, Argentina</query>
</request>New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "clicking the Next button moves you to the next screen".
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.