Book Image

Learning Pentaho Data Integration 8 CE - Third Edition

Book Image

Learning Pentaho Data Integration 8 CE - Third Edition

Overview of this book

Pentaho Data Integration(PDI) is an intuitive and graphical environment packed with drag-and-drop design and powerful Extract-Tranform-Load (ETL) capabilities. This book shows and explains the new interactive features of Spoon, the revamped look and feel, and the newest features of the tool including transformations and jobs Executors and the invaluable Metadata Injection capability. We begin with the installation of PDI software and then move on to cover all the key PDI concepts. Each of the chapter introduces new features, enabling you to gradually get practicing with the tool. First, you will learn to do all kind of data manipulation and work with simple plain files. Then, the book teaches you how you can work with relational databases inside PDI. Moreover, you will be given a primer on data warehouse concepts and you will learn how to load data in a data warehouse. During the course of this book, you will be familiarized with its intuitive, graphical and drag-and-drop design environment. By the end of this book, you will learn everything you need to know in order to meet your data manipulation requirements. Besides, your will be given best practices and advises for designing and deploying your projects.

Related Content you might be interested in

Current Title:

Small book image
Learning Pentaho Data Integration 8 CE - Third Edition
Dec, 2017 | 500 pages
Small book image
Pentaho 8 Reporting for Java Developers
Francesco Corti
Sep, 2017 | 470 pages
Table of Contents (23 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Pentaho Data Integration
Getting Started with Pentaho Data Integration
Pentaho Data Integration and Pentaho BI Suite
Installing PDI
Launching the PDI Graphical Designer - Spoon
Introducing transformations
Installing useful related software
Summary
2
Getting Started with Transformations
Getting Started with Transformations
Designing and previewing transformations
Understanding PDI data and metadata
Handling errors
Summary
3
Creating Basic Task Flows
Creating Basic Task Flows
Introducing jobs
Designing and running jobs
Running transformations from a Job
Understanding and changing the flow of execution
Managing files
Knowing the basics about Kettle variables
Summary
4
Reading and Writing Files
Reading and Writing Files
Reading data from files
Outputting data to files
Working with Big Data and cloud sources
Summary
5
Manipulating PDI Data and Metadata
Manipulating PDI Data and Metadata
Manipulating simple fields
Working with complex structures
Summary
6
Controlling the Flow of Data
Controlling the Flow of Data
Filtering data
Splitting streams unconditionally
Splitting the stream based on conditions
Merging streams in several ways
Looking up data
Summary
7
Cleansing, Validating, and Fixing Data
Cleansing, Validating, and Fixing Data
Cleansing data
Validating data
Treating invalid data by splitting and merging streams
Summary
8
Manipulating Data by Coding
Manipulating Data by Coding
Doing simple tasks with the JavaScript step
Parsing unstructured files with JavaScript
Doing simple tasks with the Java Class step
Getting the most out of the Java Class step
Avoiding coding using purpose-built steps
Summary
9
Transforming the Dataset
Transforming the Dataset
Sorting data
Working on groups of rows
Converting rows to columns
Normalizing data
Going forward and backward across rows
Summary
10
Performing Basic Operations with Databases
Performing Basic Operations with Databases
Connecting to a database and exploring its content
Previewing and getting data from a database
Inserting, updating, and deleting data
Verifying a connection, running DDL scripts, and doing other useful tasks
Looking up data in different ways
Summary
11
Loading Data Marts with PDI
Loading Data Marts with PDI
Preparing the environment
Introducing dimensional modeling
Loading dimensions with data
Loading fact tables
Summary
12
Creating Portable and Reusable Transformations
Creating Portable and Reusable Transformations
Defining and using Kettle variables
Creating reusable Transformations
Making the data flow between transformations
Executing transformations in an iterative way
Summary
13
Implementing Metadata Injection
Implementing Metadata Injection
Introducing metadata injection
Discovering metadata and injecting it
Identifying use cases to implement metadata injection
Summary
14
Creating Advanced Jobs
Creating Advanced Jobs
Enhancing your processes with the use of variables
Accessing copied rows for different purposes
Working with filelists
Executing jobs in an iterative way
Summary
15
Launching Transformations and Jobs from the Command Line
Launching Transformations and Jobs from the Command Line
Using the Pan and Kitchen utilities
Supplying named parameters and variables
Using command-line arguments
Sending the output of executions to log files
Automating the execution
Summary
16
Best Practices for Designing and Deploying a PDI Project
Best Practices for Designing and Deploying a PDI Project
Setting up a new project
Best practices to design jobs and transformations
Maximizing the performance
Deploying the project in different environments
Summary

Setting up a new project

When you start a new PDI project, there are several aspects that we have to take into account. In this section, we will talk about the following:

  • Setting up of the environment
  • Defining a folder structure for the project
  • Availability of resources
  • Versioning

Setting up the local environment

When you run Spoon for the first time, a folder named .kettle is created by default in your home directory. This folder is referred to as the Kettle home directory. As you already know, it's in this folder where the kettle.properties file is located. The folder contains additional configuration files, which are summarized in the following table:

FILE

DESCRIPTION

kettle.properties

Definition of Kettle variables with Java Virtual Machine scope.

shared.xml

Definition of shared objects as for example database connections.

.spoonrc

User interface settings. This file contains the settings that you defined in the Tool | Options... setting window in Spoon, as well as the history of your work, for example...

Previous Section
End of Section 2
Next Section