Databricks Data Engineer Professional Question 276
Single answerYou are managing a Databricks job that processes a large dataset using multiple tasks in a task graph. One of the tasks has failed due to an intermittent issue with a data source. The task is idempotent, and you want to rerun only the failed task without reprocessing tasks that have already completed successfully. What is the most appropriate way to repair and rerun the failed task in Databricks?
- A
Use the 'Repair and Rerun' option in the Databricks job run UI and select the failed task to rerun.
- B
Delete the job and create a new job with the same configuration to ensure a fresh run.
- C
Manually rerun all tasks in the job, starting from the first task, to ensure data consistency.
- D
Modify the job cluster configuration and restart the entire job run.
Show answer and explanation
Correct answer: A
Explanation
Databricks provides a 'Repair and Rerun' feature that allows you to rerun only the failed tasks in a job run without reprocessing successfully completed tasks. This feature is particularly useful for idempotent tasks and ensures efficient use of resources. Other options, such as restarting the entire job or creating a new job, are not necessary and can lead to wasted compute resources.
- A. Correct.
This is correct. The 'Repair and Rerun' option in the Databricks job run UI allows you to rerun only the failed task(s) in a job without reprocessing completed tasks, which is efficient and preserves resources.
- B. Incorrect.
This is incorrect. Deleting the job and creating a new one is unnecessary and inefficient for handling a failed task. Databricks provides built-in features for handling such scenarios.
- C. Incorrect.
This is incorrect. Manually rerunning all tasks in the job is not resource-efficient and defeats the purpose of using Databricks' repair functionality.
- D. Incorrect.
This is incorrect. Modifying the job cluster configuration does not address the root problem of rerunning a specific failed task and could introduce additional complications.