Google Professional Data Engineer Question 83
Select 2Google Cloud PlatformYou are building a data pipeline to process streaming data from IoT sensors using Google Cloud. The pipeline must ensure low latency, scalability, and fault tolerance. Which combination of components should you consider when designing this pipeline?
- A
Cloud Pub/Sub for message ingestion and Cloud Dataflow for stream processing
- B
Cloud SQL for storing raw IoT data and BigQuery for analytics
- C
Cloud Storage for long-term archival and Dataflow for data transformation
- D
Cloud Pub/Sub for message ingestion and Cloud Functions for lightweight event processing
- E
Dataproc for real-time stream processing and Bigtable for storing time-series data
Show answer and explanation
Correct answers: A, D
Explanation
The correct combination of tools for a streaming data pipeline must handle ingestion, processing, and scalability efficiently. Cloud Pub/Sub is a managed service designed for real-time message ingestion and delivery, while Cloud Dataflow provides a unified model for both stream and batch processing. Alternatively, Cloud Pub/Sub paired with Cloud Functions can handle lightweight processing needs with minimal operational complexity.
- A. Correct.
Cloud Pub/Sub is an ideal tool for message ingestion in a streaming pipeline, and Cloud Dataflow provides robust, scalable stream processing with low latency. This combination is well-suited for the scenario.
- B. Incorrect.
While Cloud SQL can store data, it is not optimized for ingesting large-scale streaming IoT data. BigQuery is better suited for analytics but not for real-time ingestion or processing.
- C. Incorrect.
Cloud Storage is suitable for archival purposes, but it cannot handle real-time stream processing. Dataflow can transform data, but it requires an appropriate ingestion mechanism like Pub/Sub.
- D. Correct.
Cloud Pub/Sub is an excellent choice for message ingestion, and Cloud Functions can handle lightweight event processing. This combination is scalable and works well for specific scenarios requiring low operational overhead.
- E. Incorrect.
Dataproc is designed for batch processing or heavy computation workloads, not real-time stream processing. Bigtable is a good choice for storing time-series data but does not handle ingestion or processing.