Google Professional Data Engineer Question 111
Select 3Google Cloud PlatformYou are designing a data pipeline in Google Cloud for a retail company. The pipeline ingests real-time customer transaction data using Pub/Sub, processes the data using Dataflow, and stores the output in BigQuery for analytics. As part of operationalizing this pipeline, you need to ensure it is reliable, cost-efficient, and easy to monitor. Which of the following practices should you implement?
- A
Enable Dataflow Auto-scaling to optimize resource usage during fluctuating workloads.
- B
Set up Pub/Sub Dead Letter Topics to handle message delivery failures.
- C
Use manual Dataflow worker node allocation to maintain predictable costs.
- D
Activate BigQuery Slot Reservations to control query execution costs.
- E
Implement Cloud Monitoring alerts to notify the team of pipeline errors or performance issues.
Show answer and explanation
Correct answers: A, B, E
Explanation
Operationalizing a data pipeline involves ensuring reliability, cost efficiency, and effective monitoring. Practices like enabling Dataflow Auto-scaling, setting up Pub/Sub Dead Letter Topics, and implementing Cloud Monitoring alerts directly address these needs. Manual worker node allocation and BigQuery Slot Reservations are less relevant or inefficient for this scenario.
- A. Correct.
Enabling Dataflow Auto-scaling dynamically adjusts the number of worker nodes to handle changes in workload, optimizing both cost and performance. This is a recommended practice for operational efficiency.
- B. Correct.
Setting up Pub/Sub Dead Letter Topics ensures that undeliverable messages are logged and can be analyzed later, improving reliability and fault tolerance in the pipeline.
- C. Incorrect.
Manual Dataflow worker node allocation can lead to underutilization or over-provisioning of resources, making it less efficient than auto-scaling for fluctuating workloads.
- D. Incorrect.
Activating BigQuery Slot Reservations is useful for managing costs in environments with predictable, high-volume queries, but it is not directly relevant to operationalizing real-time pipelines.
- E. Correct.
Cloud Monitoring alerts help teams quickly identify and address errors or performance issues in the pipeline, improving reliability and maintainability.