Google Professional Data Engineer Question 20
Select 4Google Cloud PlatformYou are designing a data pipeline on Google Cloud to process sensitive financial data. The pipeline must support high availability, ensure data integrity throughout the processing stages, and minimize the risk of data loss. Which actions should you take to design for reliability and fidelity?
- A
Implement a multi-region storage strategy using Cloud Storage buckets with versioning enabled.
- B
Use Pub/Sub with message acknowledgment and dead-letter topics to handle message failures.
- C
Store intermediate pipeline results only in local SSDs to maximize processing speed.
- D
Enable Stackdriver (Cloud Monitoring) alerts to detect anomalies in pipeline performance.
- E
Design the pipeline to retry failed operations with exponential backoff.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Designing for reliability and fidelity in data pipelines involves implementing strategies to ensure high availability, integrity, and fault tolerance. Multi-region storage, Pub/Sub with acknowledgment mechanisms, monitoring tools like Stackdriver, and retry mechanisms with backoff are all critical components for achieving these goals. Local SSDs, while fast, do not provide durability and should not be used for critical intermediate storage.
- A. Correct.
Correct: A multi-region storage strategy ensures high availability and durability, and enabling versioning provides a safeguard for accidental overwrites or deletions.
- B. Correct.
Correct: Pub/Sub with message acknowledgment and dead-letter topics ensures reliable message delivery and allows for failure handling, maintaining data integrity.
- C. Incorrect.
Incorrect: Storing intermediate results only on local SSDs risks data loss if a node fails, as local SSDs are not persistent storage.
- D. Correct.
Correct: Enabling Stackdriver alerts helps detect and respond to pipeline anomalies, improving reliability.
- E. Correct.
Correct: Retrying failed operations with exponential backoff ensures that transient issues do not cause permanent failures, enhancing reliability and fidelity.