Google Professional Data Engineer Question 274
Select 3Google Cloud PlatformYour organization uses Google Cloud for its data processing workflows. Recently, a key analytics pipeline experienced issues due to missing data and potential data corruption. As a Professional Data Engineer, how should you design the pipeline to proactively handle these issues and ensure reliable data processing?
- A
Implement validation checks at each stage of the pipeline to detect anomalies or missing data.
- B
Enable Cloud Pub/Sub Dead Letter Topics for capturing messages that fail processing.
- C
Use BigQuery's streaming insert API without retries to minimize delays in data ingestion.
- D
Configure Cloud Dataflow to use built-in windowing and watermarking to handle late or missing data.
- E
Rely solely on downstream systems to handle data quality issues instead of addressing them within the pipeline.
Show answer and explanation
Correct answers: A, B, D
Explanation
To prepare for data corruption and missing data, it is essential to design pipelines that proactively validate data, handle errors gracefully, and incorporate features like Dead Letter Topics and windowing to deal with incomplete or late data. This approach ensures data integrity and reliability across the workflow.
- A. Correct.
Validation checks ensure that missing or corrupt data is detected early and can be handled appropriately, reducing downstream issues.
- B. Correct.
Cloud Pub/Sub Dead Letter Topics allow you to capture failed messages for further analysis or reprocessing, minimizing data loss.
- C. Incorrect.
While BigQuery's streaming insert API is useful, disabling retries can lead to increased data loss, which is contrary to the goal of ensuring data reliability.
- D. Correct.
Cloud Dataflow's windowing and watermarking features allow you to manage late data and ensure that incomplete datasets are handled gracefully.
- E. Incorrect.
Relying solely on downstream systems to handle data quality issues can lead to inefficiencies and increased complexity, as issues should ideally be addressed closer to their source.