Summary
In this chapter, we've learned about streaming data and we learned how to handle incoming data as soon as data is created. Data is created using the Pub/Sub publisher client. In practice, you can use this approach by requesting the application developer to send messages to Pub/Sub as the data source, or a second option is to use a change data capture (CDC) tool. In GCP, you can use a Google provided tool for CDC called Datastream. CDC tools can be attached to the backend database like CloudSQL to publish data changes such as insert, update, and delete operations. We as data engineers are responsible for using Pub/Sub, as we've learned in this chapter.
The second part of streaming data is how to process the data. In this chapter, we've learned how to use Dataflow to handle continuously incoming data from Pub/Sub to aggregate it on the fly and store it in BigQuery tables. Do keep in mind that you can also handle data from Pub/Sub using Dataflow in batch manner...