Book Image

Learning Pentaho Data Integration 8 CE - Third Edition

Book Image

Learning Pentaho Data Integration 8 CE - Third Edition

Overview of this book

Pentaho Data Integration(PDI) is an intuitive and graphical environment packed with drag-and-drop design and powerful Extract-Tranform-Load (ETL) capabilities. This book shows and explains the new interactive features of Spoon, the revamped look and feel, and the newest features of the tool including transformations and jobs Executors and the invaluable Metadata Injection capability. We begin with the installation of PDI software and then move on to cover all the key PDI concepts. Each of the chapter introduces new features, enabling you to gradually get practicing with the tool. First, you will learn to do all kind of data manipulation and work with simple plain files. Then, the book teaches you how you can work with relational databases inside PDI. Moreover, you will be given a primer on data warehouse concepts and you will learn how to load data in a data warehouse. During the course of this book, you will be familiarized with its intuitive, graphical and drag-and-drop design environment. By the end of this book, you will learn everything you need to know in order to meet your data manipulation requirements. Besides, your will be given best practices and advises for designing and deploying your projects.

Related Content you might be interested in

Current Title:

Small book image
Learning Pentaho Data Integration 8 CE - Third Edition
Dec, 2017 | 500 pages
Small book image
Pentaho 8 Reporting for Java Developers
Francesco Corti
Sep, 2017 | 470 pages
Table of Contents (23 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Pentaho Data Integration
Getting Started with Pentaho Data Integration
Pentaho Data Integration and Pentaho BI Suite
Installing PDI
Launching the PDI Graphical Designer - Spoon
Introducing transformations
Installing useful related software
Summary
2
Getting Started with Transformations
Getting Started with Transformations
Designing and previewing transformations
Understanding PDI data and metadata
Handling errors
Summary
3
Creating Basic Task Flows
Creating Basic Task Flows
Introducing jobs
Designing and running jobs
Running transformations from a Job
Understanding and changing the flow of execution
Managing files
Knowing the basics about Kettle variables
Summary
4
Reading and Writing Files
Reading and Writing Files
Reading data from files
Outputting data to files
Working with Big Data and cloud sources
Summary
5
Manipulating PDI Data and Metadata
Manipulating PDI Data and Metadata
Manipulating simple fields
Working with complex structures
Summary
6
Controlling the Flow of Data
Controlling the Flow of Data
Filtering data
Splitting streams unconditionally
Splitting the stream based on conditions
Merging streams in several ways
Looking up data
Summary
7
Cleansing, Validating, and Fixing Data
Cleansing, Validating, and Fixing Data
Cleansing data
Validating data
Treating invalid data by splitting and merging streams
Summary
8
Manipulating Data by Coding
Manipulating Data by Coding
Doing simple tasks with the JavaScript step
Parsing unstructured files with JavaScript
Doing simple tasks with the Java Class step
Getting the most out of the Java Class step
Avoiding coding using purpose-built steps
Summary
9
Transforming the Dataset
Transforming the Dataset
Sorting data
Working on groups of rows
Converting rows to columns
Normalizing data
Going forward and backward across rows
Summary
10
Performing Basic Operations with Databases
Performing Basic Operations with Databases
Connecting to a database and exploring its content
Previewing and getting data from a database
Inserting, updating, and deleting data
Verifying a connection, running DDL scripts, and doing other useful tasks
Looking up data in different ways
Summary
11
Loading Data Marts with PDI
Loading Data Marts with PDI
Preparing the environment
Introducing dimensional modeling
Loading dimensions with data
Loading fact tables
Summary
12
Creating Portable and Reusable Transformations
Creating Portable and Reusable Transformations
Defining and using Kettle variables
Creating reusable Transformations
Making the data flow between transformations
Executing transformations in an iterative way
Summary
13
Implementing Metadata Injection
Implementing Metadata Injection
Introducing metadata injection
Discovering metadata and injecting it
Identifying use cases to implement metadata injection
Summary
14
Creating Advanced Jobs
Creating Advanced Jobs
Enhancing your processes with the use of variables
Accessing copied rows for different purposes
Working with filelists
Executing jobs in an iterative way
Summary
15
Launching Transformations and Jobs from the Command Line
Launching Transformations and Jobs from the Command Line
Using the Pan and Kitchen utilities
Supplying named parameters and variables
Using command-line arguments
Sending the output of executions to log files
Automating the execution
Summary
16
Best Practices for Designing and Deploying a PDI Project
Best Practices for Designing and Deploying a PDI Project
Setting up a new project
Best practices to design jobs and transformations
Maximizing the performance
Deploying the project in different environments
Summary

Reading data from files

Despite being the most primitive format used to store data, files are still broadly used and they exist in several formats, such as fixed width, comma-separated values, spreadsheets, or even free format files. Pentaho Data Integration (PDI) has the ability to read data from all kinds of files. In this section, let's see how to use PDI to get data from these files.

Reading a simple file

In this section, you will learn to read one of the most common input sources, plain files.

For demonstration purposes, we will use a simplified version of sales_data.csv that comes with the PDI bundle. Our sample file looks as follows:

ORDERDATE,ORDERNUMBER,ORDERLINENUMBER,PRODUCTCODE,PRODUCTLINE,QUANTITYORDERED,PRICEEACH,SALES
2/20/2004 0:00 ,10223,10,S24_4278 ,Planes ,23,74.62,1716.26
11/21/2004 0:00,10337,3,S18_4027 ,Classic Cars ,36,100 ,5679.36
6/16/2003 0:00 ,10131,2,S700_4002,Planes ,26,85.13,2213.38
7/6/2004 0:00 ,10266,5,S18_1984 ,Classic Cars ,49,100 ,6203.4
10/16/2004 0:00,10310...
Previous Section
End of Section 2
Next Section