Google Professional Data Engineer Question 269
Select 3Google Cloud PlatformYou are designing a streaming data processing system on Google Cloud that ingests real-time sensor data from IoT devices. The system must be highly fault-tolerant to ensure minimal data loss and should automatically restart in case of failures. Which of the following design choices would help you achieve this requirement?
- A
Use Google Cloud Dataflow with checkpointing enabled.
- B
Deploy the system on a single Compute Engine instance with local disk storage.
- C
Enable Pub/Sub message retention and dead-letter topics.
- D
Use Google Cloud Storage for intermediate data storage.
- E
Implement retries and exponential backoff for failed operations in your application logic.
Show answer and explanation
Correct answers: A, C, E
Explanation
To design a fault-tolerant system for streaming data processing, it is essential to implement checkpointing in your data processing pipeline (e.g., in Dataflow), ensure that message retention and error handling are configured (e.g., in Pub/Sub), and include retries with exponential backoff for transient failures. These strategies collectively ensure minimal data loss and support automatic recovery in case of failures. Avoid single points of failure, such as using a single Compute Engine instance with local disk storage, as it compromises fault tolerance.
- A. Correct.
Enabling checkpointing in Google Cloud Dataflow ensures that the system can resume processing from the last checkpoint in case of a failure, making it a critical component for fault tolerance.
- B. Incorrect.
Using a single Compute Engine instance with local disk storage creates a single point of failure and does not provide fault tolerance or automatic restarts.
- C. Correct.
Enabling Pub/Sub message retention and configuring dead-letter topics ensures that messages are not lost in case of processing failures. This is a key design choice for fault tolerance in streaming systems.
- D. Incorrect.
Google Cloud Storage is suitable for storing large datasets, but it is not optimized for real-time fault tolerance or restart scenarios in streaming data processing systems.
- E. Correct.
Retries with exponential backoff are a best practice for handling transient errors, ensuring that failed operations can be retried without overwhelming the system.