Google Professional Data Engineer Question 222
Select 3Google Cloud PlatformYou are managing a data pipeline on Google Cloud that processes a high volume of streaming data in real-time. The pipeline uses Dataflow for ETL operations and BigQuery for analytics. Lately, you have noticed an increase in costs without a corresponding increase in performance. Which actions should you take to optimize resource usage and reduce costs while maintaining performance?
- A
Use Dataflow's autoscaling feature to automatically adjust the number of workers based on workload.
- B
Switch to using on-demand pricing for BigQuery instead of flat-rate pricing.
- C
Enable partitioning and clustering for BigQuery tables to minimize the amount of data scanned during queries.
- D
Reduce Dataflow's worker machine types to smaller instances to save on compute costs.
- E
Configure Dataflow jobs to use Streaming Engine to decouple compute and storage for better scalability.
Show answer and explanation
Correct answers: A, C, E
Explanation
To optimize resources and reduce costs while maintaining performance, it's crucial to leverage features that align with scalable and efficient resource management. Autoscaling in Dataflow dynamically adjusts resources based on the workload, partitioning and clustering in BigQuery minimizes data scanned, and using the Streaming Engine in Dataflow enhances scalability by decoupling compute and storage. These approaches ensure that costs are aligned with actual usage while maintaining high performance.
- A. Correct.
Using Dataflow's autoscaling feature ensures that the number of workers scales up or down based on the actual workload, which can optimize both performance and cost.
- B. Incorrect.
Switching to on-demand pricing for BigQuery may not reduce costs if queries are frequent or predictable. Flat-rate pricing is often more cost-effective for consistent, high-volume workloads.
- C. Correct.
Enabling partitioning and clustering in BigQuery reduces the amount of data scanned during queries, thereby optimizing both performance and cost.
- D. Incorrect.
Reducing the machine type to smaller instances may lead to under-provisioning, which can negatively impact performance and increase latency in processing.
- E. Correct.
Using the Streaming Engine in Dataflow decouples storage and compute, allowing for better scalability and resource optimization without compromising performance.