Google Professional Data Engineer Question 21
Select 3Google Cloud PlatformYou are designing a data processing pipeline for a retail analytics platform that must handle high levels of traffic during seasonal sales events. The pipeline must ensure data integrity, handle sudden spikes in data volume, and provide high availability. Which of the following design considerations would help you meet these requirements?
- A
Implement autoscaling for the data processing infrastructure.
- B
Use a managed database service with built-in replication.
- C
Store data temporarily in a single-region storage bucket to optimize costs.
- D
Enable retries and exponential backoff for failed data ingestion requests.
- E
Run the pipeline on a single virtual machine to simplify architecture.
Show answer and explanation
Correct answers: A, B, D
Explanation
To design for reliability and fidelity in a data processing pipeline, it is essential to consider scalability, replication, and fault-tolerance mechanisms. Autoscaling ensures the system can handle traffic spikes, replication ensures data availability and integrity, and retries with exponential backoff address transient failures. Single-region storage and single-machine setups compromise reliability and availability, making them unsuitable for such scenarios.
- A. Correct.
Implementing autoscaling allows the infrastructure to handle sudden spikes in data volume by dynamically adding or removing resources based on demand, ensuring reliability.
- B. Correct.
Using a managed database service with built-in replication ensures high availability and data integrity by automatically replicating data across multiple zones or regions.
- C. Incorrect.
Storing data in a single-region storage bucket might reduce costs but introduces a reliability risk if the region experiences an outage, making it a poor choice for high availability requirements.
- D. Correct.
Enabling retries and exponential backoff for failed requests ensures that transient errors during data ingestion do not result in data loss, improving reliability and data integrity.
- E. Incorrect.
Running the pipeline on a single virtual machine creates a single point of failure, reducing both reliability and availability.