Google Professional Data Engineer Question 240
Single answerGoogle Cloud PlatformYou are a Data Engineer responsible for managing a batch job that processes user activity logs every day at midnight. The data needs to be uploaded to BigQuery for analysis. Which solution should you implement to schedule this job in a repeatable and reliable way?
- A
Use Cloud Scheduler to trigger a Cloud Function that runs the data processing job and uploads the results to BigQuery.
- B
Use a cron job on a Compute Engine instance to execute a script that processes the data and uploads it to BigQuery.
- C
Manually run the data processing job every day at midnight.
- D
Use Dataflow's built-in scheduling capabilities to schedule the job at midnight.
Show answer and explanation
Correct answer: A
Explanation
The best approach is to use Cloud Scheduler, a managed service provided by Google Cloud, to schedule jobs in a repeatable and reliable manner. Cloud Scheduler can trigger various services, including Cloud Functions, to execute workflows. This ensures that the job runs on schedule without requiring manual intervention or additional maintenance.
- A. Correct.
This is correct because Cloud Scheduler is designed for reliable and repeatable scheduling of jobs. It can trigger a Cloud Function to execute the job and upload the data to BigQuery.
- B. Incorrect.
This is not the best option because cron jobs on Compute Engine lack the reliability and scalability of managed services like Cloud Scheduler. They require additional maintenance and monitoring.
- C. Incorrect.
This is incorrect because manually running the job every day is not repeatable or reliable. It introduces the risk of human error and missed schedules.
- D. Incorrect.
This is incorrect because Dataflow does not have built-in scheduling capabilities. You would need to use an external service like Cloud Scheduler to trigger Dataflow jobs at specific times.