Google Professional Data Engineer Question 158
Single answerGoogle Cloud PlatformYour organization processes large batches of transactional data daily to generate sales reports. The data is stored in Cloud Storage as JSON files. You need to implement a scalable and cost-effective solution to process this data and produce aggregated results, such as total sales per region, which will be stored in BigQuery for analytics. Which of the following solutions should you use?
- A
Use Dataflow with a streaming pipeline to process the JSON files as they are uploaded to Cloud Storage.
- B
Use Dataflow with a batch pipeline to read the JSON files from Cloud Storage, process the data, and write the results to BigQuery.
- C
Use Dataproc with a Spark job to read the JSON files from Cloud Storage, process the data, and write the results to BigQuery.
- D
Use a Cloud Function triggered by Cloud Storage to process the JSON files and write the results directly to BigQuery.
Show answer and explanation
Correct answer: B
Explanation
Dataflow batch pipelines are ideal for processing large daily datasets, as they provide scalability, cost-efficiency, and integration with BigQuery. Streaming pipelines and event-driven solutions like Cloud Functions are not appropriate for this scenario, while Dataproc introduces unnecessary complexity for this use case.
- A. Incorrect.
Dataflow streaming pipelines are suitable for continuous or real-time data processing, but this scenario involves processing large daily batches, making a batch pipeline more appropriate and cost-effective.
- B. Correct.
Dataflow batch pipelines are specifically designed for processing large datasets in a scalable and cost-effective manner. This matches the requirement to process daily batches of JSON files and store aggregated results in BigQuery.
- C. Incorrect.
Dataproc with Spark can be used for batch processing, but it often requires more configuration and maintenance compared to Dataflow. Dataflow is more integrated with GCP and better suited for this use case.
- D. Incorrect.
Cloud Functions are designed for lightweight, event-driven tasks and are not ideal for processing large datasets or generating complex aggregated results.