Google Professional Data Engineer Question 160
Select 2Google Cloud PlatformYour company processes large volumes of streaming data from IoT devices and needs to calculate real-time metrics such as the average temperature per device every minute. The processed data should also be stored for further batch analysis. Which Google Cloud tools should you use to implement this pipeline?
- A
Cloud Pub/Sub to ingest data and Dataflow to process streaming data in real-time.
- B
BigQuery to calculate the real-time metrics and store the results.
- C
Cloud Storage to store the data temporarily and Dataproc for real-time processing.
- D
BigQuery to store the processed data and perform batch analysis on it.
- E
Dataflow to ingest data and Cloud Spanner to process real-time metrics.
Show answer and explanation
Correct answers: A, D
Explanation
The correct solution involves using Cloud Pub/Sub to ingest real-time streaming data and Dataflow to process the data. BigQuery is then used to store the processed results and perform batch analysis. This setup aligns with Google Cloud best practices for handling real-time streaming and batch analysis requirements efficiently.
- A. Correct.
Cloud Pub/Sub is a highly scalable messaging service suitable for ingesting streaming data, and Dataflow provides real-time stream processing capabilities, making this an appropriate option.
- B. Incorrect.
BigQuery is not optimized for real-time metric calculations but is excellent for storing processed data and performing batch analysis, so this option is partially correct.
- C. Incorrect.
Cloud Storage is not intended for real-time processing; it is better suited for batch or archival storage. Dataproc is primarily used for Hadoop and Spark workloads, not real-time stream processing.
- D. Correct.
BigQuery is well-suited for storing processed data and batch analysis, making this a correct option for the batch analysis requirement.
- E. Incorrect.
Dataflow can handle real-time processing, but Cloud Spanner is a transactional database and is not designed for calculating real-time metrics, making this option incorrect.