Book Image

Connecting the Data: Data Integration Techniques for Building an Operational Data Store (ODS)

By : Angelo Bobak
Book Image

Connecting the Data: Data Integration Techniques for Building an Operational Data Store (ODS)

By: Angelo Bobak

Overview of this book

When organizations change or enhance their internal structures, business data integration is a complex problem that they must resolve. This book describes the common hurdles you might face while working with data integration and shows you various ways to overcome these challenges. The book begins by explaining the foundational concepts of ODS. Once familiar with schema integration, you?ll learn how to reverse engineer each data source for creating a set of data dictionary reports. These reports will provide you with the metadata necessary to apply the schema integration process. As you progress through the chapters, you will learn how to write scripts for populating the source databases and spreadsheets, as well as how to use reports to create Extract, Transform, and Load (ETL) specifications. By the end of the book, you will have the knowledge necessary to design and build a small ODS.
Table of Contents (17 chapters)
Free Chapter
1
Section 1: Site Reliability Engineering – A Prescriptive Way to Implement DevOps
6
Section 2: Google Cloud Services to Implement DevOps via CI/CD
Appendix: Getting Ready for Professional Cloud DevOps Engineer Certification

Eliminating toil through automation

Toil was introduced in Chapter 1, DevOps, SRE, and Google Cloud Services for CI/CD, and is defined as the work tied to a production service where the characteristic of that work is manual, repetitive, automatable, tactical, lacks enduring value, and linearly grows with the service. Toil is often confused with overhead, but overhead refers to administrative work that includes email, commute, filing expense reports, and attending meetings. Toil can be both good and bad—it really depends on the amount of toil.

Here are some of the positive sides of performing toil, but in very short and limited amounts:

  • Produces a sense of gratification or a sense of accomplishment
  • Can act as a low-stress or low-risk activity
  • Can be used to train new recruits, especially providing them a chance to learn by being hands-on with the system to learn the inner workings

However, excessive toil can lead to the following problems or issues...