Google Professional Data Engineer Question 113
Select 3Google Cloud PlatformYou are designing a data processing pipeline on Google Cloud for a real-time analytics application. The pipeline ingests data from Cloud Pub/Sub, processes it using Dataflow, and stores the output in BigQuery. To ensure the pipeline is reliable and operationally efficient, which of the following actions should you take during deployment and operationalization?
- A
Configure Cloud Monitoring and set up custom alerts for pipeline performance metrics.
- B
Use the default Dataflow autoscaling settings without additional tuning.
- C
Implement retries and dead-letter queues for failed messages in Dataflow.
- D
Enable BigQuery's table expiration settings to manage storage costs.
- E
Disable Dataflow's job logs to reduce operational overhead.
Show answer and explanation
Correct answers: A, C, D
Explanation
To deploy and operationalize a data pipeline effectively, it is important to ensure observability, reliability, and cost management. Cloud Monitoring with alerts enables proactive issue resolution, while retries and dead-letter queues ensure data processing reliability. BigQuery's table expiration settings help control storage costs. Default configurations (like Dataflow autoscaling) may need tuning, and disabling logs is not recommended as it reduces visibility into pipeline operations.
- A. Correct.
Correct. Configuring Cloud Monitoring and setting up alerts helps ensure the pipeline can be monitored effectively and issues can be addressed promptly.
- B. Incorrect.
Incorrect. While Dataflow's default autoscaling is useful, tuning it based on workload characteristics often improves performance and cost efficiency.
- C. Correct.
Correct. Implementing retries and dead-letter queues is essential for handling failed messages and ensuring data processing reliability.
- D. Correct.
Correct. Enabling BigQuery's table expiration settings helps manage storage costs by automatically deleting unused data.
- E. Incorrect.
Incorrect. Disabling Dataflow's job logs can hinder debugging and operational monitoring, which is critical for maintaining pipeline reliability.