Google Professional Data Engineer Question 239
Single answerGoogle Cloud PlatformYou are a Data Engineer at a retail company that processes sales data daily. You need to set up a job in Google Cloud to process the sales data from a Cloud Storage bucket and load it into BigQuery every day at midnight. Which approach should you use to schedule this job in a repeatable and scalable way?
- A
Use Cloud Scheduler to trigger a Cloud Function that processes the data and loads it into BigQuery.
- B
Manually initiate the job using the BigQuery web UI every day at midnight.
- C
Set up a cron job on a Compute Engine instance to schedule the BigQuery data load.
- D
Use Cloud Composer and define a Directed Acyclic Graph (DAG) to schedule the data processing job.
Show answer and explanation
Correct answer: A
Explanation
Cloud Scheduler is the most appropriate solution for scheduling jobs in a repeatable way in Google Cloud. It provides a lightweight, serverless way to trigger tasks at defined intervals, such as daily at midnight. In this scenario, Cloud Scheduler can trigger a Cloud Function to process the data and load it into BigQuery, achieving automation without requiring the overhead of managing infrastructure or complex workflows.
- A. Correct.
This is the correct answer. Cloud Scheduler is designed for scheduling tasks in a repeatable way. It can trigger a Cloud Function, which can be used to process data from Cloud Storage and load it into BigQuery efficiently.
- B. Incorrect.
This option is not scalable or automated, as it requires manual intervention every day, which defeats the purpose of repeatable and automated scheduling.
- C. Incorrect.
While cron jobs on Compute Engine can schedule tasks, this approach is less scalable, requires maintaining infrastructure, and is not the recommended method for scheduling jobs in Google Cloud.
- D. Incorrect.
Although Cloud Composer is a good option for managing complex workflows, it is overkill for a simple daily job like this. Cloud Scheduler is a more lightweight and appropriate solution.