Google Professional Data Engineer Question 84
Select 3Google Cloud PlatformYou are designing a data pipeline in Google Cloud for processing streaming data from IoT devices. The pipeline must handle high-throughput data ingestion, process data in near real-time, and ensure scalability and fault tolerance. Which of the following components should you use to build this pipeline?
- A
Cloud Pub/Sub for ingesting streaming data
- B
Dataflow for real-time data processing
- C
Cloud SQL for high-throughput data storage
- D
BigQuery for analytical queries on processed data
- E
Dataproc for batch processing of historical data
Show answer and explanation
Correct answers: A, B, D
Explanation
To design a scalable, fault-tolerant pipeline for real-time streaming data, Cloud Pub/Sub is used for ingesting data, Dataflow for processing data in real-time, and BigQuery for analytical queries on the processed data. Cloud SQL and Dataproc are not suitable choices for this specific use case as they are not optimized for real-time, high-throughput streaming workloads.
- A. Correct.
Cloud Pub/Sub is a fully managed messaging service that is highly scalable and suitable for ingesting high-throughput streaming data.
- B. Correct.
Dataflow is ideal for real-time data processing and supports scalable, fault-tolerant stream processing.
- C. Incorrect.
Cloud SQL is not designed for high-throughput data ingestion or real-time processing; it is better suited for transactional databases.
- D. Correct.
BigQuery is a serverless data warehouse optimized for analytical queries, making it suitable for querying processed data in this scenario.
- E. Incorrect.
Dataproc is designed for batch processing of large datasets, which is not relevant for the real-time streaming requirements of this pipeline.