Google Professional Data Engineer Question 93
Single answerGoogle Cloud PlatformYou are working as a data engineer for a retail company that processes large volumes of transaction data. The data is generated throughout the day but is not required to be processed in real time. Instead, reports are generated nightly for business analysis. Which Google Cloud service and approach would best suit this use case for batch processing?
- A
Use Cloud Dataflow with a streaming pipeline to process data as it arrives
- B
Use Cloud Dataproc to run scheduled Apache Spark jobs for nightly data processing
- C
Use BigQuery with scheduled queries to process and aggregate the data at night
- D
Use Cloud Pub/Sub to push transaction data in real-time to downstream systems
Show answer and explanation
Correct answer: B
Explanation
The requirement specifies batch processing of transaction data to generate nightly reports. Cloud Dataproc is ideal for this use case as it allows you to create and schedule jobs for processing large data sets using tools like Apache Spark or Hadoop, which are commonly used for batch workloads. Other options are either designed for real-time processing or better suited for querying existing data rather than processing it in batches.
- A. Incorrect.
Cloud Dataflow with a streaming pipeline is designed for real-time data processing, which is unnecessary for this use case as the reports are only generated nightly.
- B. Correct.
Cloud Dataproc is a suitable choice for this batch processing use case, as it supports running scheduled Apache Spark or Hadoop jobs to process large amounts of data at a specified time.
- C. Incorrect.
While BigQuery can schedule queries, it is more suited for querying and analyzing data stored in BigQuery tables rather than processing raw data as part of a batch workflow.
- D. Incorrect.
Cloud Pub/Sub is used for real-time messaging and event-driven architectures. It does not directly handle batch processing or reporting requirements.