Getting hands-on – a big data case study
An unnamed company wants to build a data analytics pipeline for their IoT data and has turned to you for guidance on running a proof of concept on GCP. This company (your client) has a team of analysts who already work with Apache Beam, and they wish to keep using the same framework to avoid a steep learning curve. Your client's IoT devices produce semi-structured data, and they also want to have a data warehouse solution for storing all of it. They expect the number of devices and telemetry data generated to scale nearly exponentially as they expand in the next few years, so they want to ensure that the solution is highly scalable and future-proof.
You come up with the following design decisions:
To prepare a proof of concept, you then perform the following steps:
- Go to the GCP console (console.cloud.google.com), and then click on the shell icon in the top-right corner of the screen to activate...