Google Professional Data Engineer Question 109
Single answerGoogle Cloud PlatformYour organization recently adopted Google Cloud for data processing and analytics. You have been tasked with integrating a new data source that provides real-time IoT sensor data using MQTT. The goal is to analyze this data in near real-time and store it for future use. Which of the following approaches would best integrate this new data source into your data pipeline?
- A
Use Cloud Pub/Sub to ingest the MQTT data, process it with Dataflow, and store it in BigQuery for analytics.
- B
Set up a Cloud Storage bucket to collect the MQTT data directly, then use Dataproc to process the data before storing it in BigQuery.
- C
Deploy an MQTT broker on Compute Engine, write custom Python code to process the data, and store the results in a Firestore database.
- D
Use Bigtable as the direct ingestion point for MQTT data and run periodic queries to analyze the data.
Show answer and explanation
Correct answer: A
Explanation
When integrating a new streaming data source like MQTT, the best practice is to use Google Cloud services optimized for real-time data ingestion and processing. Cloud Pub/Sub acts as a scalable messaging service to ingest the data, Dataflow processes it in near real-time, and BigQuery serves as a powerful analytics platform to analyze and store the data. This approach leverages Google Cloud's managed services to minimize operational overhead and maximize performance.
- A. Correct.
This is the correct answer. Cloud Pub/Sub is designed to handle real-time streaming data, making it a suitable choice for ingesting MQTT messages. Dataflow can process the data in near real-time, and BigQuery is an excellent choice for analytics and querying.
- B. Incorrect.
This approach is suboptimal because Cloud Storage is not designed for real-time ingestion of streaming data like MQTT. Dataproc is more suited for batch processing rather than real-time analysis.
- C. Incorrect.
While deploying an MQTT broker on Compute Engine is possible, managing it and writing custom code introduces significant operational overhead. Firestore is not optimized for analytical queries, making this approach less efficient for the given use case.
- D. Incorrect.
Bigtable is a high-throughput, low-latency database, but it is not designed as an ingestion point for streaming data like MQTT. Additionally, Bigtable is not the best choice for ad-hoc analytics compared to BigQuery.