Google Professional Data Engineer Question 88
Select 2Google Cloud PlatformYour organization is building a real-time data processing pipeline to analyze streaming logs from IoT devices. The pipeline must support high-throughput, low-latency message ingestion and enable the transformation of streaming data for storage in BigQuery for analytics. Which combination of Google Cloud services should you use to design this pipeline?
- A
Cloud Pub/Sub for message ingestion and Dataflow for stream processing
- B
Dataproc running Apache Kafka for message ingestion and Apache Spark for stream processing
- C
Cloud Data Fusion for batch processing and Cloud Pub/Sub for message ingestion
- D
BigQuery for message ingestion and Cloud Dataflow for stream processing
- E
Cloud Pub/Sub for message ingestion and BigQuery for direct stream processing
Show answer and explanation
Correct answers: A, B
Explanation
For a real-time streaming pipeline, Cloud Pub/Sub and Dataflow are the most suitable fully managed services for high-throughput, low-latency message ingestion and stream processing. Alternatively, for organizations using open-source tools, Dataproc with Apache Kafka and Apache Spark can also achieve the same functionality but requires additional management and configuration. Both combinations align with the requirements of the scenario.
- A. Correct.
Cloud Pub/Sub is a fully managed messaging service designed for real-time message ingestion, and Dataflow is ideal for stream processing using Apache Beam. This combination supports high-throughput and low-latency requirements for streaming pipelines.
- B. Correct.
Dataproc allows the use of Apache Kafka and Apache Spark, which are open-source tools for message ingestion and stream processing, respectively. This setup is suitable for organizations with existing Hadoop ecosystems but requires more management effort compared to fully managed services.
- C. Incorrect.
Cloud Data Fusion is primarily used for batch data integration and transformation, not for real-time stream processing. While Cloud Pub/Sub is suitable for message ingestion, this combination does not meet the low-latency requirements of the scenario.
- D. Incorrect.
BigQuery is a data warehouse and is not designed for message ingestion. While Cloud Dataflow can process streaming data, this combination does not provide a real-time message ingestion mechanism.
- E. Incorrect.
While Cloud Pub/Sub is appropriate for message ingestion, BigQuery is not designed for direct stream processing. Data must first be processed and transformed before being ingested into BigQuery.