Google Professional Data Engineer Question 225
Select 2Google Cloud PlatformYour organization processes large volumes of streaming data from IoT devices, and the business requires near real-time analytics. The team has decided to use Google Cloud services for this use case, but you are tasked with minimizing costs while meeting the performance requirements. Which combination of services and configurations would best meet the needs?
- A
Use Cloud Pub/Sub for ingesting streaming data and Dataflow with autoscaling enabled for real-time processing.
- B
Use BigQuery for storing and analyzing the data with a flat-rate pricing model for consistent costs.
- C
Use Cloud Pub/Sub for ingesting streaming data and a Cloud Dataflow job with fixed worker instances to process data.
- D
Use Bigtable for storing streaming data and querying it with Dataflow for analytics.
- E
Use Dataproc with preemptible VMs for ingesting and processing streaming data.
Show answer and explanation
Correct answers: A, B
Explanation
To minimize costs while meeting the need for near real-time analytics, Cloud Pub/Sub and Dataflow provide an efficient ingestion and processing pipeline, especially with Dataflow's autoscaling feature. BigQuery complements this setup as a cost-effective and scalable analytics platform. Other options either do not meet the real-time requirement, are suboptimal for analytics, or do not offer cost-effective scaling for the use case.
- A. Correct.
This is a cost-effective and scalable solution. Cloud Pub/Sub is designed for low-latency message ingestion, and enabling autoscaling on Dataflow ensures that the number of resources adjusts dynamically based on workload, keeping costs low.
- B. Correct.
BigQuery is a cost-effective and serverless analytics solution for storing and querying large datasets. The flat-rate pricing model provides cost predictability, which is suitable for businesses with consistent workloads.
- C. Incorrect.
Using a fixed number of worker instances in Dataflow might not optimize costs as workload demands can vary, potentially leading to either over-provisioning (higher costs) or under-provisioning (performance issues).
- D. Incorrect.
While Bigtable is optimized for low-latency, high-throughput workloads, it is not ideal for analytics use cases on streaming data as it lacks built-in querying capabilities like BigQuery.
- E. Incorrect.
Dataproc with preemptible VMs can lower costs, but it is more suitable for batch processing rather than real-time streaming analytics, which is the requirement in this scenario.