Google Professional Data Engineer Question 273
Select 3Google Cloud PlatformYou are designing a data pipeline on Google Cloud for a retail company that ingests and processes real-time transaction data. The company wants to ensure resilience against data corruption and missing data during ingestion and processing. Which of the following approaches should you implement?
- A
Implement Cloud Pub/Sub with dead-letter topics to capture messages that fail to process.
- B
Use schema validation in Cloud Dataflow to ensure that the data adheres to the expected format before processing.
- C
Configure BigQuery to automatically retry failed queries multiple times to handle transient errors.
- D
Create a Cloud Storage bucket with Object Versioning enabled to retain previous versions of overwritten or deleted data.
- E
Disable Cloud Logging for performance optimization to avoid capturing irrelevant logs.
Show answer and explanation
Correct answers: A, B, D
Explanation
To prepare for data corruption and missing data, it's important to implement mechanisms for detecting, capturing, and recovering from such issues. Pub/Sub dead-letter topics allow for capturing failed messages, schema validation ensures early detection of data issues, and Cloud Storage Object Versioning provides a way to recover overwritten or deleted data. These practices collectively improve the resilience of the data pipeline.
- A. Correct.
Correct: Cloud Pub/Sub dead-letter topics ensure that messages that fail processing due to corruption or other issues are captured for later analysis or reprocessing, improving resilience.
- B. Correct.
Correct: Schema validation in Cloud Dataflow ensures that malformed or missing data is flagged early in the pipeline, preventing downstream errors.
- C. Incorrect.
Incorrect: While retrying BigQuery queries can handle transient errors during querying, it does not address data corruption or missing data in the pipeline.
- D. Correct.
Correct: Enabling Object Versioning in Cloud Storage allows recovery of previous versions of files, which is helpful in case of accidental overwrites or deletions, mitigating data loss.
- E. Incorrect.
Incorrect: Disabling Cloud Logging may hinder debugging and auditing efforts, reducing visibility into data issues, which is counterproductive for resilience.