Databricks Data Engineer Associate Question 446
Single answerA data engineering team is tasked with running a job in Databricks to aggregate daily sales data and store it in a Delta table. The job must run automatically at 3:00 AM every day. Which approach should the team use to schedule this job in Databricks?
- A
Create a Databricks job and configure a CRON expression to run it daily at 3:00 AM.
- B
Manually trigger the job daily at 3:00 AM using the Databricks CLI.
- C
Use an external workflow orchestrator like Apache Airflow to schedule the job with a CRON expression.
- D
Create a Python script to run the job and use a sleep loop to ensure it triggers at 3:00 AM daily.
Show answer and explanation
Correct answer: A
Explanation
Databricks jobs provide built-in scheduling capabilities using CRON expressions, making them the most efficient and reliable solution for automating recurring tasks like running a job at a specific time daily. By configuring a CRON expression in Databricks, the team can schedule the job to run at 3:00 AM without relying on external tools or manual intervention.
- A. Correct.
This is the correct approach. Databricks jobs support CRON expressions for scheduling, allowing users to automate job execution at specific times (e.g., 3:00 AM daily).
- B. Incorrect.
This is not a practical solution for automating jobs since it requires manual intervention every day, which is not feasible for recurring tasks.
- C. Incorrect.
Although external orchestrators like Apache Airflow can use CRON expressions, it is unnecessary to rely on an external tool when Databricks jobs natively support CRON scheduling.
- D. Incorrect.
Using a sleep loop in a Python script is not a reliable or efficient way to schedule jobs. It lacks the robustness and precision of a proper scheduling mechanism like CRON.