Book Image

Pentaho Data Integration Beginner's Guide - Second Edition - Second Edition

By : María Carina Roldán
Book Image

Pentaho Data Integration Beginner's Guide - Second Edition - Second Edition

By: María Carina Roldán

Overview of this book

Capturing, manipulating, cleansing, transferring, and loading data effectively are the prime requirements in every IT organization. Achieving these tasks require people devoted to developing extensive software programs, or investing in ETL or data integration tools that can simplify this work. Pentaho Data Integration is a full-featured open source ETL solution that allows you to meet these requirements. Pentaho Data Integration has an intuitive, graphical, drag-and-drop design environment and its ETL capabilities are powerful. However, getting started with Pentaho Data Integration can be difficult or confusing. "Pentaho Data Integration Beginner's Guide - Second Edition" provides the guidance needed to overcome that difficulty, covering all the possible key features of Pentaho Data Integration. "Pentaho Data Integration Beginner's Guide - Second Edition" starts with the installation of Pentaho Data Integration software and then moves on to cover all the key Pentaho Data Integration concepts. Each chapter introduces new features, allowing you to gradually get involved with the tool. First, you will learn to do all kinds of data manipulation and work with plain files. Then, the book gives you a primer on databases and teaches you how to work with databases inside Pentaho Data Integration. Moreover, you will be introduced to data warehouse concepts and you will learn how to load data in a data warehouse. After that, you will learn to implement simple and complex processes. Finally, you will have the opportunity of applying and reinforcing all the learned concepts through the implementation of a simple datamart. With "Pentaho Data Integration Beginner's Guide - Second Edition", you will learn everything you need to know in order to meet your data manipulation requirements.
Table of Contents (26 chapters)
Pentaho Data Integration Beginner's Guide
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Best Practices
Index

Appendix A. Working with Repositories

Spoon allows you to store your transformations and jobs under two different configurations: file-based and database repository. In contrast to the file-based configuration that keeps the transformations and jobs in XML format as *.ktr and *.kjb files in the local filesystem, the database repository configuration keeps the same information in tables in a relational database.

Although working with the file-based system is simple and practical, the database repository method can be convenient in some situations.

The following is a list of some of the distinctive repository features:

  • Repositories implement security. In order to work with a repository, you need credentials.

  • Repositories are, by their nature, prepared for basic team development. The elements you create (transformations, jobs, database connections, and so on) are shared by all the repository users as soon as you create them.

  • The Enterprise Repository is a Java content repository capable of more robust and scalable collaborative functions such as version control, locking, and more.

Before you decide on working with a repository, you have to be aware of the file-based system benefits you lose. Here are some examples:

  • When working with the database repository-based system, you need access to the repository database. If for some reason you cannot access it (for example, network problems), you will not be able to work. You don't have this restriction when working with files, where you only need the software and the transformation and job files, that is, the .ktr and .kjb files.

  • When working with the database repositories, it is difficult to keep track of the changes. Working with the filesystem, it is easier to know which jobs or transformations were modified. If you use Subversion or Git, you even have a control version that allows you to examine the history of changes and to recover older versions of your work if necessary.

  • Suppose that you want to search and replace some text in all the jobs and transformations. If you are working with repositories, you would have to do it for each table in the repository database. Whereas working with the file-based system, this task is quite simple. For example, you could create a Sublime project - available for downloading at www.sublimetext.com - open the root directory of your jobs and transformations, and do the task by using the Sublime utilities.

As explained in Chapter 1, Getting Started with Pentaho Data Integration, there is a third method, File repository, that is a mix of the two mentioned earlier. It's a repository of jobs and transformations stored in the filesystem.

Note

The use of the File repository is similar to the database repository. Therefore, we will not explain it in this appendix. You should not have any difficulty in trying it once you understand how to work with the database repository.

This appendix shows you how to create a database repository and how to work with it. You can try repositories and decide for yourself which method, database repository-based or file-based, suits you best.