In this section, we will learn how to set up a Dataproc cluster, submit a job to the cluster, and do some interesting things with Dataproc.
- Navigate to the project where you wish to set up the cluster.
- Click on the menu button in the top-left corner with three horizontal lines and choose Dataproc from the drop-down menu:
- Click on the Create Cluster button and fill in the required information for the cluster, which is cluster name, memory and CPU configurations, region, GPU, and disk configurations. We will name our cluster my-cluster and choose single node globally:
- To configure the cluster further, you can visit the staging bucket for the cluster, which is used to store configurations and control files. The staging buckets are separate for each region, so make sure your staging bucket is in the same region as your cluster. To view it, use the following...