Book Image

Learning Pentaho Data Integration 8 CE - Third Edition

Book Image

Learning Pentaho Data Integration 8 CE - Third Edition

Overview of this book

Pentaho Data Integration(PDI) is an intuitive and graphical environment packed with drag-and-drop design and powerful Extract-Tranform-Load (ETL) capabilities. This book shows and explains the new interactive features of Spoon, the revamped look and feel, and the newest features of the tool including transformations and jobs Executors and the invaluable Metadata Injection capability. We begin with the installation of PDI software and then move on to cover all the key PDI concepts. Each of the chapter introduces new features, enabling you to gradually get practicing with the tool. First, you will learn to do all kind of data manipulation and work with simple plain files. Then, the book teaches you how you can work with relational databases inside PDI. Moreover, you will be given a primer on data warehouse concepts and you will learn how to load data in a data warehouse. During the course of this book, you will be familiarized with its intuitive, graphical and drag-and-drop design environment. By the end of this book, you will learn everything you need to know in order to meet your data manipulation requirements. Besides, your will be given best practices and advises for designing and deploying your projects.

Related Content you might be interested in

Current Title:

Small book image
Learning Pentaho Data Integration 8 CE - Third Edition
Dec, 2017 | 500 pages
Small book image
Pentaho 8 Reporting for Java Developers
Francesco Corti
Sep, 2017 | 470 pages
Table of Contents (23 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Pentaho Data Integration
Getting Started with Pentaho Data Integration
Pentaho Data Integration and Pentaho BI Suite
Installing PDI
Launching the PDI Graphical Designer - Spoon
Introducing transformations
Installing useful related software
Summary
2
Getting Started with Transformations
Getting Started with Transformations
Designing and previewing transformations
Understanding PDI data and metadata
Handling errors
Summary
3
Creating Basic Task Flows
Creating Basic Task Flows
Introducing jobs
Designing and running jobs
Running transformations from a Job
Understanding and changing the flow of execution
Managing files
Knowing the basics about Kettle variables
Summary
4
Reading and Writing Files
Reading and Writing Files
Reading data from files
Outputting data to files
Working with Big Data and cloud sources
Summary
5
Manipulating PDI Data and Metadata
Manipulating PDI Data and Metadata
Manipulating simple fields
Working with complex structures
Summary
6
Controlling the Flow of Data
Controlling the Flow of Data
Filtering data
Splitting streams unconditionally
Splitting the stream based on conditions
Merging streams in several ways
Looking up data
Summary
7
Cleansing, Validating, and Fixing Data
Cleansing, Validating, and Fixing Data
Cleansing data
Validating data
Treating invalid data by splitting and merging streams
Summary
8
Manipulating Data by Coding
Manipulating Data by Coding
Doing simple tasks with the JavaScript step
Parsing unstructured files with JavaScript
Doing simple tasks with the Java Class step
Getting the most out of the Java Class step
Avoiding coding using purpose-built steps
Summary
9
Transforming the Dataset
Transforming the Dataset
Sorting data
Working on groups of rows
Converting rows to columns
Normalizing data
Going forward and backward across rows
Summary
10
Performing Basic Operations with Databases
Performing Basic Operations with Databases
Connecting to a database and exploring its content
Previewing and getting data from a database
Inserting, updating, and deleting data
Verifying a connection, running DDL scripts, and doing other useful tasks
Looking up data in different ways
Summary
11
Loading Data Marts with PDI
Loading Data Marts with PDI
Preparing the environment
Introducing dimensional modeling
Loading dimensions with data
Loading fact tables
Summary
12
Creating Portable and Reusable Transformations
Creating Portable and Reusable Transformations
Defining and using Kettle variables
Creating reusable Transformations
Making the data flow between transformations
Executing transformations in an iterative way
Summary
13
Implementing Metadata Injection
Implementing Metadata Injection
Introducing metadata injection
Discovering metadata and injecting it
Identifying use cases to implement metadata injection
Summary
14
Creating Advanced Jobs
Creating Advanced Jobs
Enhancing your processes with the use of variables
Accessing copied rows for different purposes
Working with filelists
Executing jobs in an iterative way
Summary
15
Launching Transformations and Jobs from the Command Line
Launching Transformations and Jobs from the Command Line
Using the Pan and Kitchen utilities
Supplying named parameters and variables
Using command-line arguments
Sending the output of executions to log files
Automating the execution
Summary
16
Best Practices for Designing and Deploying a PDI Project
Best Practices for Designing and Deploying a PDI Project
Setting up a new project
Best practices to design jobs and transformations
Maximizing the performance
Deploying the project in different environments
Summary

Maximizing the performance

A PDI Transformation may be colorful, organized, full of pretty notes, but at the end, the most important thing is that it meets its purpose within a reasonable time of execution. Therefore, it's important to test your work with real data in order to detect if there are things that can be affecting the performance.

There are many factors involved both internal—for example, bad configuration of sort steps—and external—for example lack of indexes in a database table. Fortunately, PDI offers some means to analyze and improve the performance of your transformations.

Analyzing Steps Metrics

You are already familiar with the Step Metrics tab, which shows statistics for each step in your Transformation. There are some columns that we haven't mentioned until now, which are useful to monitor the performance:

STEP METRIC

DESCRIPTION

Time

The time of execution of the step

Speed (r/s)

The processing speed in rows per second

input/output

The number of rows in the input and output buffer...

Previous Section
End of Section 4
Next Section