Book Image

Connecting the Data: Data Integration Techniques for Building an Operational Data Store (ODS)

By : Angelo Bobak
Book Image

Connecting the Data: Data Integration Techniques for Building an Operational Data Store (ODS)

By: Angelo Bobak

Overview of this book

When organizations change or enhance their internal structures, business data integration is a complex problem that they must resolve. This book describes the common hurdles you might face while working with data integration and shows you various ways to overcome these challenges. The book begins by explaining the foundational concepts of ODS. Once familiar with schema integration, you?ll learn how to reverse engineer each data source for creating a set of data dictionary reports. These reports will provide you with the metadata necessary to apply the schema integration process. As you progress through the chapters, you will learn how to write scripts for populating the source databases and spreadsheets, as well as how to use reports to create Extract, Transform, and Load (ETL) specifications. By the end of the book, you will have the knowledge necessary to design and build a small ODS.
Table of Contents (17 chapters)
Free Chapter
1
Section 1: Site Reliability Engineering – A Prescriptive Way to Implement DevOps
6
Section 2: Google Cloud Services to Implement DevOps via CI/CD
Appendix: Getting Ready for Professional Cloud DevOps Engineer Certification

Managing build artifacts – Container Registry

Source code management is the first step in the CI process. This is followed by building the code. Code can be built based on various trigger points; either against a development branch or when a PR is merged into the master branch. The code build process can result in one or more artifacts. Based on the nature of the code being built, the resultant artifacts can either be binaries, packages, container images, or a combination. These artifacts are stored in a registry and then deployed into a computing environment and form the CD process. In between the CI and CD process, there is an intermediate process where the build artifacts are stored and then subsequently deployed. This is known as artifact management.

Artifact management acts as a single source of truth and a critical integration point between CI and CD. Many artifact management systems provide versioning, the ability to scan for vulnerabilities, provide consistent configuration...