Databricks Data Engineer Professional Question 275
Single answerA Databricks job that processes large datasets has failed due to an intermittent connection issue when writing to an external database. The job consists of multiple tasks, and only the last task failed. The job is configured with task dependencies and incremental checkpointing enabled. What is the most appropriate action to repair and rerun the failed job?
- A
Delete the job run and rerun the entire job from the beginning.
- B
Use the 'Repair and Rerun' feature in the Jobs UI to restart only the failed task and its dependent tasks.
- C
Manually identify the failed task, fix the issue, and rerun the entire job manually.
- D
Modify the job's configuration to ignore task dependencies and rerun the entire job.
Show answer and explanation
Correct answer: B
Explanation
The 'Repair and Rerun' feature in the Databricks Jobs UI is specifically designed to handle scenarios where only certain tasks in a job fail. It ensures that only the failed task and its dependent tasks are rerun while maintaining the state of successfully completed tasks using incremental checkpointing. This approach is efficient and avoids unnecessary recomputation of successful tasks.
- A. Incorrect.
Deleting and rerunning the entire job would unnecessarily reprocess tasks that have already succeeded, wasting compute resources.
- B. Correct.
Using the 'Repair and Rerun' feature in the Jobs UI allows you to rerun only the failed task and any dependent tasks, leveraging the incremental checkpointing to avoid reprocessing tasks that were successful.
- C. Incorrect.
Manually identifying and fixing the issue is not required in this case because Databricks provides a built-in feature to repair and rerun failed tasks automatically.
- D. Incorrect.
Ignoring task dependencies and rerunning the job could lead to inconsistent results and data integrity issues, especially if tasks require dependencies to execute correctly.