Databricks Data Engineer Professional Question 277
Select 3You are a data engineer tasked with creating a Databricks Job to process daily sales data from a Delta table and generate a summary report. The job should execute a notebook that performs data aggregation and writes the results back to another Delta table. Additionally, the job must be scheduled to run every day at midnight. Which of the following configurations would you use while creating this Databricks Job?
- A
Set the job type to 'Notebook' and specify the path to the notebook containing the aggregation logic.
- B
Configure the cluster type as a high-concurrency cluster for better performance and concurrent task execution.
- C
Set the job schedule to run once at midnight and specify a recurring interval of 24 hours.
- D
Enable retries in the job configuration with a maximum retry count to handle transient failures.
- E
Add a task dependency to ensure the job waits for another upstream job to complete before running.
Show answer and explanation
Correct answers: A, C, D
Explanation
To create a reliable Databricks Job for processing daily sales data, you need to configure the job to execute a notebook containing the aggregation logic, schedule it to run daily at midnight, and enable retries to handle potential transient failures. Task dependencies and high-concurrency clusters are not relevant in this specific scenario, making options 1, 3, and 4 the correct answers.
- A. Correct.
Setting the job type to 'Notebook' and specifying the notebook path is essential as the notebook contains the logic for processing the data.
- B. Incorrect.
Using a high-concurrency cluster is not required for this scenario as the job is processing data in a single, scheduled run. A standard or job cluster may be more appropriate.
- C. Correct.
Setting the schedule to run at midnight with a 24-hour interval ensures the job executes daily at the desired time.
- D. Correct.
Enabling retries in the job configuration is a best practice to handle transient failures and improve reliability of the job.
- E. Incorrect.
Adding a task dependency is not relevant in this scenario as there is no mention of an upstream job that needs to complete before this job runs.