Google Professional Data Engineer Question 94
Single answerGoogle Cloud PlatformYou are a data engineer working for a retail company that processes large volumes of daily sales data. The data is stored in Google Cloud Storage and needs to be aggregated daily to generate sales reports, which are then stored in BigQuery. The aggregation process can tolerate delays of several hours, and the reports are not required in real-time. Which approach would be the most appropriate for this use case?
- A
Use a batch processing pipeline with Dataflow to read data from Cloud Storage, process it, and write the results to BigQuery.
- B
Use a streaming pipeline with Dataflow to continuously process incoming data from Cloud Storage and write the results to BigQuery.
- C
Use Cloud Functions to trigger a data processing job each time a new file is uploaded to Cloud Storage and store the results in BigQuery.
- D
Use Dataproc to run a Spark job on a schedule to process the data from Cloud Storage and write the results to BigQuery.
Show answer and explanation
Correct answer: A
Explanation
The scenario describes a use case that involves processing large volumes of data on a daily basis, with no real-time requirements. A batch processing solution like Dataflow is ideal for this because it is designed to handle such workloads efficiently, scales seamlessly, and integrates natively with both Cloud Storage and BigQuery. Streaming solutions or event-driven approaches are unnecessary and may introduce unnecessary complexity or inefficiency.
- A. Correct.
This is the most appropriate option because batch processing with Dataflow is well-suited for aggregating large volumes of data that do not require real-time results. Dataflow provides scalable and efficient processing.
- B. Incorrect.
Streaming pipelines are designed for real-time or near-real-time use cases, which is unnecessary in this scenario since the processing can tolerate delays of several hours.
- C. Incorrect.
While Cloud Functions could be used to trigger processing jobs, this approach may not handle large-scale batch processing efficiently and is better suited for event-driven, lightweight tasks.
- D. Incorrect.
Dataproc can handle batch processing, but it requires managing a cluster and is more suitable for specialized tasks that involve distributed computation frameworks like Hadoop or Spark, which is unnecessary for this use case.