Google Professional Data Engineer Question 95
Single answerGoogle Cloud PlatformYou are working as a Data Engineer at a retail company. They want to process daily sales data uploaded to a Cloud Storage bucket in CSV format. The processed data will be used for daily reporting, and the system must handle large files efficiently. Which approach should you take to implement this batch processing pipeline?
- A
Use Cloud Dataflow to read the CSV files from Cloud Storage, process the data, and write the output to BigQuery.
- B
Use BigQuery to directly query the raw CSV files in Cloud Storage without any preprocessing.
- C
Use a Cloud Function triggered by a file upload to process the CSV file and write the results back to Cloud Storage.
- D
Use Pub/Sub to stream the data from the uploaded CSV files for real-time processing.
Show answer and explanation
Correct answer: A
Explanation
Cloud Dataflow is the ideal choice for batch processing in this scenario. It can handle large file processing efficiently, supports transformations, and integrates seamlessly with Cloud Storage and BigQuery. This makes it suitable for daily reporting needs in a batch processing pipeline.
- A. Correct.
This is the correct approach. Cloud Dataflow is specifically designed for scalable batch and stream processing. It can efficiently handle large files, process the data, and write the results to BigQuery for reporting.
- B. Incorrect.
BigQuery can query external CSV files in Cloud Storage, but this approach is inefficient for large-scale data and may not provide the required processing capabilities for transforming the data.
- C. Incorrect.
Cloud Functions are not suitable for processing large files as they have execution time and memory limits. They are better suited for lightweight, event-driven tasks.
- D. Incorrect.
Pub/Sub is a messaging service designed for real-time stream processing, not for batch processing of large files.