Google Professional Data Engineer Question 69
Select 3Google Cloud PlatformYou are tasked with designing a data pipeline to process large volumes of real-time streaming data from IoT devices. The data must be ingested, processed, and stored in near real-time for analysis. Which of the following considerations should you prioritize when planning the pipeline?
- A
Ensuring the pipeline can scale horizontally to handle increasing data volumes
- B
Minimizing upfront costs by using fixed-size infrastructure
- C
Selecting a storage system optimized for low-latency writes and reads
- D
Ensuring data integrity by implementing exactly-once processing semantics
- E
Using batch processing for all data to simplify pipeline design
Show answer and explanation
Correct answers: A, C, D
Explanation
When designing a real-time data pipeline for IoT devices, it is crucial to focus on scalability, low-latency storage, and data integrity to ensure the pipeline can handle large volumes of streaming data efficiently and accurately. Fixed-size infrastructure and batch processing are not suitable for real-time use cases.
- A. Correct.
Correct: Horizontal scalability is critical for real-time pipelines dealing with increasing data volumes, as it ensures that the system can handle growth without performance degradation.
- B. Incorrect.
Incorrect: While minimizing costs is important, fixed-size infrastructure is not suitable for a scalable real-time pipeline, as it limits the ability to handle variable or growing workloads.
- C. Correct.
Correct: Real-time processing requires a storage system capable of low-latency writes and reads to ensure efficient data handling and timely analysis.
- D. Correct.
Correct: Maintaining data integrity through exactly-once processing semantics is essential for avoiding data duplication or loss in real-time pipelines.
- E. Incorrect.
Incorrect: Batch processing is not suitable for real-time scenarios, as it introduces delays and does not meet the near real-time requirement.