Google Professional Data Engineer Question 67
Select 3Google Cloud PlatformYour organization needs to design a data pipeline to process streaming data from IoT devices. The pipeline must ensure low-latency processing, support scalability for fluctuating data volumes, and handle potential data loss during transmission. Which considerations should you prioritize while planning the data pipeline?
- A
Choose a stream processing framework that supports exactly-once processing semantics.
- B
Implement data sharding to distribute the workload evenly across resources.
- C
Use batch processing instead of streaming to simplify pipeline design.
- D
Design for fault-tolerance to handle potential node or network failures.
- E
Rely on a fixed set of resources regardless of data volume fluctuations.
Show answer and explanation
Correct answers: A, B, D
Explanation
When designing a data pipeline for streaming IoT data, it's essential to prioritize low-latency processing, scalability, and reliability. Exactly-once processing semantics ensure data consistency, data sharding enables scalability, and fault-tolerance ensures the pipeline can recover from failures. Batch processing and fixed resource allocation do not meet the requirements for handling streaming data efficiently.
- A. Correct.
This is correct. Choosing a stream processing framework that supports exactly-once processing semantics ensures data accuracy and consistency in a streaming pipeline, which is critical for IoT data processing.
- B. Correct.
This is correct. Implementing data sharding improves scalability by distributing the workload evenly across multiple processing units, making the pipeline more efficient under varying data loads.
- C. Incorrect.
This is incorrect. Batch processing is not suitable for streaming data use cases as it introduces significant latency, which contradicts the low-latency requirement.
- D. Correct.
This is correct. Fault-tolerance is critical for ensuring the availability and reliability of the pipeline during network or hardware failures.
- E. Incorrect.
This is incorrect. Relying on a fixed set of resources is not scalable and would lead to performance bottlenecks during high data volume periods.