Book Image

Google Cloud Platform Cookbook

By : Legorie Rajan PS
Book Image

Google Cloud Platform Cookbook

By: Legorie Rajan PS

Overview of this book

Google Cloud Platform is a cloud computing platform that offers products and services to host applications using state-of-the art infrastructure and technology. You can build and host applications and websites, store data, and analyze data on Google's scalable infrastructure. This book follows a recipe-based approach, giving you hands-on experience to make the most of Google Cloud services. This book starts with practical recipes that explain how to utilize Google Cloud's common services. Then, you'll see how to make full use of Google Cloud components such as networking, security, management, and developer tools. Next, we'll deep dive into implementing core Google Cloud services into your organization, with practical recipes on App Engine, Compute Engine, Cloud Functions, virtual networks, and Cloud Storage. Later, we'll provide recipes on implementing authentication and security, Cloud APIs, command-line management, deployment management, and the Cloud SDK. Finally, we'll cover administration and troubleshooting tasks on applications with Compute services and we'll show how to monitor your organization's efficiency with best practices. By the end of this book, you'll have an overall understanding and hands-on implementation of Google Cloud services in your organization with ease.
Table of Contents (14 chapters)
Title Page
Dedication
Packt Upsell
Contributors
Preface
Index

Creating a Dataflow pipeline to store streaming data


Google Dataflow is a service for stream and batch processing at scale. When there is a need for processing lots of streamed data like click stream or data from IoT devices, Dataflow will be the starting point for receiving all the stream data. The data can then be sent to storage (BigQuery, Bigtable, GCS) for further processing (ML):

For this recipe, let's consider a weather station (IoT device) that is sending temperature data to GCP. The data is emitted constantly by the IoT device and is stored on Google Storage for later analytics processing. Considering the intermittent nature of data connectivity between the device and GCP, we'll need a solution to receive the messages, process/handle them, and store them. For this solution, we'll create a Dataflow pipeline using a Google provided template—Cloud Pub/Sub to Cloud Storage text.

Getting ready

The following are the initial setup verification steps for the creation of the network before...