Book Image

Connecting the Data: Data Integration Techniques for Building an Operational Data Store (ODS)

By : Angelo Bobak
Book Image

Connecting the Data: Data Integration Techniques for Building an Operational Data Store (ODS)

By: Angelo Bobak

Overview of this book

When organizations change or enhance their internal structures, business data integration is a complex problem that they must resolve. This book describes the common hurdles you might face while working with data integration and shows you various ways to overcome these challenges. The book begins by explaining the foundational concepts of ODS. Once familiar with schema integration, you?ll learn how to reverse engineer each data source for creating a set of data dictionary reports. These reports will provide you with the metadata necessary to apply the schema integration process. As you progress through the chapters, you will learn how to write scripts for populating the source databases and spreadsheets, as well as how to use reports to create Extract, Transform, and Load (ETL) specifications. By the end of the book, you will have the knowledge necessary to design and build a small ODS.
Table of Contents (17 chapters)
Free Chapter
1
Section 1: Site Reliability Engineering – A Prescriptive Way to Implement DevOps
6
Section 2: Google Cloud Services to Implement DevOps via CI/CD
Appendix: Getting Ready for Professional Cloud DevOps Engineer Certification

What this book covers

Chapter 1, DevOps, SRE, and Google Cloud Services for CI/CD, covers DevOps, which is a set of practices that builds, tests, and releases code in a repeatable and iterative manner. These practices are aimed to break down the metaphoric wall between development and operation teams. SRE is a prescriptive way from Google to implement DevOps that aligns incentives between the development and operations teams to build and maintain reliable systems. In addition, Google recommends a cloud-native development paradigm where complex systems are decomposed into multiple services using microservices architecture.

This chapter will cover topics that include the DevOps life cycle, the evolution of SRE, an introduction to key technical and cultural SRE practices, and the benefits of using a cloud-native development paradigm. The chapter will also introduce services on GCP to implement cloud-native development and apply SRE concepts.

Chapter 2, SRE Technical Practices – Deep Dive, covers reliability, which is the most critical feature of a service and should be aligned with business objectives. SRE prescribes specific technical practices to measure characteristics that define and track reliability. These technical practices include the Service-Level Agreement (SLA), Service-Level Objective (SLO), Service-Level Indicator (SLI), error budgets, and eliminating toil through automation.

This chapter goes in depth into SRE technical practices. This chapter will cover topics that include the blueprint for a well-defined SLA, defining reliability expectations via SLOs, understanding reliability targets and their implications, categorizing user journeys, sources to measure SLIs, exploring ways to make a service reliable by tracking error budgets, and eliminating toil through automation. The chapter concludes by walking through two scenarios where the impact of SLAs, SLOs, and error budgets are illustrated relative to the SLI being measured.

Chapter 3, Understanding Monitoring and Alerting to Target Reliability, discusses how the key to implementing SRE technical practices is to ensure that SLAs, SLOs, and SLIs are never violated. This makes it feasible to balance new feature releases and still maintain system reliability since the error budget is not exhausted. Monitoring, alerting, and time series are fundamental concepts to track SRE technical practices.

This chapter goes in depth into monitoring, alerting, and time series. This chapter will cover topics that include monitoring sources, monitoring types, monitoring strategies, the golden signals that are recommended to be measured and monitored, potential approaches and key attributes to define an alerting strategy, and the structure and cardinality of time series.

Chapter 4, Building SRE Teams and Applying Cultural Practices, covers how for a long time, Google considered SRE as their secret sauce to achieving system reliability while maintaining a balance with new feature release velocity. Google achieved this by applying a prescribed set of cultural practices such as incident management, being on call, and psychological safety. SRE cultural practices are required to implement SRE technical practices and are strongly recommended by Google for organizations that would like to start their SRE journey. In addition, Google has put forward aspects that are critical to building SRE teams along with an engagement model.

This chapter goes in depth into building SRE teams, which includes topics around different implementations of SRE teams, details around staffing SRE engineers, and insights into the SRE engagement model. This chapter will also cover topics on SRE cultural practices that include facets of effective incident management, factors to consider while being on call, and factors to overcome to foster psychological safety. The chapter concludes with a cultural practice that is aimed to reduce organizational silos by sharing vision and knowledge and by fostering collaboration.

Chapter 5, Managing Source Code Using Cloud Source Repositories, looks at how source code management is the first step in a CI flow. Code is stored in a source code repository such as GitHub or Bitbucket so that developers can continuously make code changes and the modified code is integrated into the repository. Cloud Source Repositories (CSR) is a service from Google Cloud that provides source code management through private Git repositories.

This chapter goes in depth into CSR. This chapter will cover topics that include key features of CSR, steps to create and access the repository, how to perform one-way sync from GitHub/Bitbucket to CSR, and common operations in CSR such as browsing repositories, performing universal code search, and detecting security keys. The chapter concludes with a hands-on lab that illustrates how code can be deployed in Cloud Functions by pulling code hosted from CSR.

Chapter 6, Building Code Using Cloud Build, and Pushing to Container Registry, looks at how once code is checked into a source code management system such as CSR, the next logical step in a CI flow is to build code, create artifacts, and push to a registry that can store the generated artifacts. Cloud Build is a service from Google Cloud that can build the source code whereas Container Registry is the destination where the created build artifacts are stored.

This chapter goes in depth into Cloud Build and Container Registry. This chapter will cover topics that include understanding the need for automation, processes to build and create container images, key essentials of Cloud Build, strategies to optimize the build speed, key essentials of Container Registry, the structure of Container Registry, and Container Analysis. The chapter concludes with a hands-on lab to build, create, push, and deploy a container to Cloud Run using Cloud Build triggers. This hands-on lab also illustrates a way to build a CI/CD pipeline as it includes both CI and automated deployment of containers to Cloud Run, a GCP compute option that runs containers.

Chapter 7, Understanding Kubernetes Essentials to Deploy Containerized Applications, covers Kubernetes, or K8s, which is an open source container orchestration system that can run containerized applications but requires significant effort in terms of setting up and ongoing maintenance. Kubernetes originated as an internal cluster management tool from Google; Google donated it to the Cloud Native Computing Foundation (CNCF) as an open source project in 2014.

This chapter goes in depth into K8s. This chapter will cover topics that include key features of K8s, elaboration of cluster anatomy, which includes components of the master control plane, node components, key Kubernetes objects such as Pods, Deployments, StatefulSets, DaemonSet, Job, CronJob, and Services, and critical factors that need to be considered while scheduling Pods. The chapter concludes with a deep dive into possible deployment strategies in Kubernetes, which includes Recreate, Rolling Update, Blue/Green, and Canary.

Chapter 8, Understanding GKE Essentials to Deploy Containerized Applications, covers Google Kubernetes Engine (GKE), which is a managed version of K8s; that is, an open source container orchestration system to automate application deployment, scaling, and cluster management. GKE requies less effort in terms of cluster creation and ongoing maintenance.

This chapter goes in depth into GKE. This chapter will cover topics that include GKE core features such as GKE node pools, GKE cluster configurations, GKE autoscaling, networking in GKE, which includes Pod and service networking, GKE storage options, and Cloud Operations for GKE. There are two hands-on labs in this chapter. The first hands-on lab is placed at the start of the chapter and illustrates cluster creation using Standard mode, deploying workloads, and exposing the Pod as a service. The chapter concludes with another hands-on lab like the first one but with the cluster creation mode as Autopilot.

Chapter 9, Securing the Cluster Using GKE Security Constructs, covers how securing a Kubernetes cluster is a critical part of deployment. Native Kubernetes provide some essential security features that focus on how a request being sent to the cluster is authenticated and authorized. It is also important to understand how the master plane components are secured along with the Pods running the applications. Additionally, GKE provides security features that are fundamental to harden a cluster's security.

This chapter goes in depth into GKE security features. This chapter will cover topics that include essential security patterns in Kubernetes, control plane security, and Pod security. This chapter concludes by discussing various GKE-specific security features such as GKE private clusters, container-optimized OS, shielded GKE nodes, restricting traffic among Pods using a network policy, deploying time security services via binary authorizations, and using a workload identity to access GCP services from applications running inside the GKE cluster.

Chapter 10, Exploring GCP Cloud Operations, looks at how once an application is deployed, the next critical phase of the DevOps life cycle is continuous monitoring as it provides a feedback loop to ensure the reliability of a service or a system. As previously discussed in Chapter 2, SRE Technical Practices – Deep Dive, SRE prescribes specific technical tools or practices that help in measuring characteristics that define and track reliability, such as SLAs, SLOs, SLIs, and error budgets. SRE prescribes the use of observability to track technical practices. Observability on GCP is established through Cloud Operations.

This chapter goes in depth into Cloud Operations. This chapter will cover topics that include Cloud Monitoring essentials such as workspaces, dashboards, Metrics Explorer, uptime checks, configuring alerting, and the need for a monitoring agent and access controls specific to Cloud Monitoring. The chapter will also cover topics that include Cloud Logging essentials such as audit logs with their classification, summarizing log characteristics across log buckets, logs-based metrics, access controls specific to Cloud Logging, network-based log types, and the use of logging agents. This chapter concludes by discussing various essentials tied to Cloud Debugger, Cloud Trace, and Cloud Profiler.