Google Professional Data Engineer Question 226
Select 3Google Cloud PlatformYour company runs a business-critical data processing pipeline on Google Cloud that processes high volumes of data daily. The pipeline uses Dataflow for streaming and batch data processing. Recently, you observed that during peak load times, the pipeline is unable to process all data within the SLA, causing delays. You need to ensure that the required resources are available for the pipeline to meet the SLA during high load periods. What actions should you take?
- A
Enable Dataflow Autoscaling to dynamically adjust the number of workers based on load.
- B
Configure Dataflow jobs with fixed worker counts to ensure consistent resource allocation.
- C
Implement Dataflow FlexRS (Flexible Resource Scheduling) to reduce the cost of processing high-priority jobs.
- D
Use Stackdriver Monitoring (now Cloud Monitoring) to set up alerts and monitor resource utilization for the pipeline.
- E
Increase the quota of Compute Engine resources in the project to handle potential scaling needs during peak times.
Show answer and explanation
Correct answers: A, D, E
Explanation
To ensure the pipeline meets SLAs during peak load periods, you should enable Dataflow Autoscaling, which dynamically adjusts worker resources based on load. Additionally, setting up monitoring allows you to proactively identify issues, while increasing Compute Engine resource quotas ensures sufficient capacity for scaling. Fixed worker counts and FlexRS are not suitable for addressing dynamic resource requirements in business-critical scenarios.
- A. Correct.
Enabling Dataflow Autoscaling allows the pipeline to dynamically allocate resources based on the current load, ensuring optimal performance during peak times. This is a necessary step to meet SLAs during high load periods.
- B. Incorrect.
Configuring Dataflow jobs with fixed worker counts can lead to underutilization or over-provisioning of resources, which is not ideal for handling fluctuating peak loads.
- C. Incorrect.
Dataflow FlexRS is designed for cost optimization by scheduling jobs during lower-demand periods, but it is not suitable for business-critical jobs with strict SLAs.
- D. Correct.
Using Stackdriver Monitoring (Cloud Monitoring) helps track resource utilization and identify bottlenecks, which is essential for ensuring that resources are available when needed.
- E. Correct.
Increasing the quota of Compute Engine resources ensures that autoscaling has sufficient resources to allocate during high-demand periods. Without adequate quotas, scaling may fail, leading to SLA violations.