Google Professional Data Engineer Question 64
Select 2Google Cloud PlatformYou are designing a data pipeline to ingest and process real-time streaming data from IoT devices. The pipeline must ensure low-latency processing and provide windowed aggregations of the data (e.g., a 5-minute average) before being written to BigQuery. Which combination of Google Cloud services should you choose to fulfill these requirements?
- A
Cloud Pub/Sub for ingestion, Dataflow for processing, BigQuery for storage
- B
Cloud Pub/Sub for ingestion, Dataproc for processing, Cloud Storage for storage
- C
Cloud Storage for ingestion, Dataflow for processing, BigQuery for storage
- D
Cloud Pub/Sub for ingestion, Cloud Functions for processing, BigQuery for storage
- E
Cloud Pub/Sub for ingestion, Dataflow for processing, Cloud Bigtable for storage
Show answer and explanation
Correct answers: A, E
Explanation
To meet the requirements of low-latency processing and windowed aggregations for streaming IoT data, Cloud Pub/Sub is used for real-time ingestion, Dataflow is used for stream processing and providing windowed aggregations, and either BigQuery or Cloud Bigtable can be used for storage depending on the use case. BigQuery is ideal for structured data analysis, while Cloud Bigtable is well-suited for time-series data with fast read/write requirements.
- A. Correct.
Correct: Cloud Pub/Sub is suitable for ingesting real-time streaming data, Dataflow supports low-latency stream processing and windowed aggregations, and BigQuery is ideal for storing and analyzing structured data.
- B. Incorrect.
Incorrect: While Pub/Sub is good for ingestion, Dataproc is more suited for batch processing rather than low-latency stream processing. Cloud Storage is also not the best option for real-time analytics as BigQuery would provide better querying capabilities.
- C. Incorrect.
Incorrect: Cloud Storage is not an ingestion tool for real-time streaming data. Dataflow and BigQuery are correct choices for processing and storage, respectively, but the ingestion tool is incorrect.
- D. Incorrect.
Incorrect: Cloud Functions can handle event-driven processing but is not designed for complex stream processing and windowed aggregations. Dataflow would be more appropriate in this case.
- E. Correct.
Correct: Cloud Pub/Sub is suitable for ingestion, Dataflow supports low-latency stream processing and windowed aggregations, and Cloud Bigtable is a good alternative for storing time-series or IoT data when scalable, low-latency access is required.