Google Professional Data Engineer Question 238
Single answerGoogle Cloud PlatformYour team manages a Google Cloud BigQuery dataset that requires daily data ingestion from an external source. To automate this process, you want to schedule a job that extracts data from the external source and loads it into the dataset at 2 AM every day. Which of the following approaches should you use to achieve this in a repeatable and reliable manner?
- A
Use Cloud Scheduler to trigger a Cloud Function that performs the data extraction and load process.
- B
Use a Compute Engine VM with a cron job to run the data extraction and load process.
- C
Manually run a BigQuery data load job every day at 2 AM using the Google Cloud Console.
- D
Use a Pub/Sub topic to trigger a BigQuery data load job directly on a daily basis.
Show answer and explanation
Correct answer: A
Explanation
The best solution for scheduling a job in a repeatable and reliable manner in Google Cloud is to use Cloud Scheduler. It allows you to set up time-based schedules and can easily trigger Cloud Functions, which are serverless and highly scalable. This approach eliminates the need for manual intervention or managing infrastructure, ensuring the process is automated and repeatable.
- A. Correct.
This is the correct answer. Cloud Scheduler is designed to trigger jobs at specific times, making it ideal for scheduling tasks like data extraction and loading. By combining it with a Cloud Function, you can implement a fully automated, repeatable, and serverless solution.
- B. Incorrect.
While a Compute Engine VM with a cron job could be used, it is not the best solution as it requires managing and maintaining the VM, which adds operational overhead. Google Cloud offers better serverless options for this use case.
- C. Incorrect.
Manually running a BigQuery data load job is not repeatable or automated, making it an unsuitable solution for this requirement.
- D. Incorrect.
Pub/Sub can trigger events, but it is not designed for time-based scheduling. You would still need another service, such as Cloud Scheduler, to schedule the trigger.