Google Professional Data Engineer Question 68
Select 3Google Cloud PlatformYou are designing a data pipeline on Google Cloud to ingest and process streaming data from IoT sensors. The pipeline needs to handle high message throughput, ensure low latency, and support real-time analytics. Which of the following considerations are most important when planning the data pipeline?
- A
Choosing a messaging system like Pub/Sub for reliable and scalable ingestion of streaming data.
- B
Using BigQuery as a message broker for real-time data ingestion and processing.
- C
Implementing windowing and aggregation in Dataflow for real-time analytics.
- D
Ensuring the pipeline scales automatically to handle spikes in data volume.
- E
Using Cloud Storage as the primary ingestion system for streaming sensor data.
Show answer and explanation
Correct answers: A, C, D
Explanation
When planning a data pipeline for streaming sensor data, it is essential to choose the right tools to ensure scalability, reliability, and low latency. Pub/Sub is an excellent choice for message ingestion, while Dataflow handles real-time processing with features like windowing. Automatic scaling ensures that the pipeline can adapt to changes in data volume. BigQuery and Cloud Storage, while valuable in other contexts, are not designed for the ingestion or broker role in real-time streaming pipelines.
- A. Correct.
Pub/Sub is specifically designed for reliable and scalable ingestion of streaming data, making it an ideal choice for this scenario.
- B. Incorrect.
BigQuery is not a message broker; it is a data warehouse. Using it as a message broker is not suitable for real-time ingestion and processing.
- C. Correct.
Windowing and aggregation in Dataflow are critical for real-time analytics, as they allow you to process and analyze data in near real-time.
- D. Correct.
Automatic scaling ensures that the pipeline can handle spikes in data volume without manual intervention, which is critical for streaming workloads.
- E. Incorrect.
Cloud Storage is not optimized for real-time ingestion of streaming data. It is better suited for batch processing and archival storage.