Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Data Engineering with Google Cloud Platform
  • Table Of Contents Toc
Data Engineering with Google Cloud Platform

Data Engineering with Google Cloud Platform - Second Edition

By : Adi Wijaya
4.5 (6)
close
close
Data Engineering with Google Cloud Platform

Data Engineering with Google Cloud Platform

4.5 (6)
By: Adi Wijaya

Overview of this book

The second edition of Data Engineering with Google Cloud builds upon the success of the first edition by offering enhanced clarity and depth to data professionals navigating the intricate landscape of data engineering. Beyond its foundational lessons, this new edition delves into the essential realm of data governance within Google Cloud, providing you with invaluable insights into managing and optimizing data resources effectively. Written by a Data Strategic Cloud Engineer at Google, this book helps you stay ahead of the curve by guiding you through the latest technological advancements in the Google Cloud ecosystem. You’ll cover essential aspects, from exploring Cloud Composer 2 to the evolution of Airflow 2.5. Additionally, you’ll explore how to work with cutting-edge tools like Dataform, DLP, Dataplex, Dataproc Serverless, and Datastream to perform data governance on datasets. By the end of this book, you'll be equipped to navigate the ever-evolving world of data engineering on Google Cloud, from foundational principles to cutting-edge practices.
Table of Contents (19 chapters)
close
close
Lock Free Chapter
1
Part 1: Getting Started with Data Engineering with GCP
4
Part 2: Build Solutions with GCP Components
11
Part 3: Key Strategies for Architecting Top-Notch Solutions

Building Workflows for Batch Data Loading Using Cloud Composer

In data engineering, the definition of a workflow is a set of configurations to automate tasks, jobs, and their dependencies. If we are talking about database workflow, we talk about how to automate the table creation process.

The main objects in any database system are tables, and one of the main differences between an application database and a data warehouse is the creation of tables. Compared to tables in application databases, where tables are mostly static and created to support frontend applications, tables in data warehouses are dynamic. The table is the main product as it is a collection of business logic, and data flows.

In this chapter, we will learn how to create the workflow for our data warehouse tables from Chapter 3, Building a Data Warehouse in BigQuery. We will learn how to automate the table creations using a Google Cloud Platform (GCP) service called Cloud Composer. This will include how to create...

CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Data Engineering with Google Cloud Platform
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist download Download options font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon