Google Professional Data Engineer Question 220
Select 3Google Cloud PlatformYour team is managing a data processing pipeline on Google Cloud that processes large volumes of batch data daily using Dataflow. You notice that the pipeline's resource usage spikes during certain times and remains underutilized during others, leading to increased costs. How can you optimize the resource usage for this pipeline while maintaining performance?
- A
Enable Dataflow autoscaling to dynamically adjust worker count based on load.
- B
Switch to using preemptible VMs for Dataflow workers where possible.
- C
Manually set the number of workers to a fixed high value to handle peak loads.
- D
Optimize the pipeline code to reduce data shuffling and improve parallelism.
- E
Use a higher machine type for all workers to improve processing speed.
Show answer and explanation
Correct answers: A, B, D
Explanation
To optimize resource usage in a Dataflow pipeline, enabling autoscaling ensures resources are dynamically allocated based on the workload, reducing costs during off-peak times. Using preemptible VMs lowers costs for batch jobs that can tolerate interruptions. Additionally, optimizing the pipeline code improves efficiency, reducing both resource usage and runtime. In contrast, over-provisioning workers or using unnecessarily high machine types can lead to increased costs and resource underutilization.
- A. Correct.
Enabling Dataflow autoscaling allows the pipeline to dynamically scale up or down the number of workers based on the load, optimizing resource usage and reducing costs during low-load periods.
- B. Correct.
Using preemptible VMs can significantly reduce costs as they are cheaper than standard VMs. This is particularly useful for batch jobs where interruptions can be tolerated.
- C. Incorrect.
Manually setting a high number of workers leads to resource over-provisioning during periods of low activity, increasing costs unnecessarily.
- D. Correct.
Optimizing the pipeline code to reduce data shuffling and improve parallelism improves efficiency, reducing the time and resources required for processing.
- E. Incorrect.
Using a higher machine type for all workers increases costs without necessarily addressing the optimization of resource usage. It may also lead to underutilized resources during low-load periods.