Databricks Data Engineer Professional Question 271
Single answerA data engineering team is using Databricks workflows to orchestrate a series of jobs that process and transform data. One of the jobs in the workflow failed due to a temporary issue when writing data to an external database. The team needs to repair the workflow and rerun only the failed task without re-executing the successful tasks. Which action should they take?
- A
Use the 'Repair and Rerun' option in the workflow UI, ensuring only the failed task is selected for re-execution.
- B
Manually delete the output of the failed task and restart the entire workflow from the beginning.
- C
Clone the workflow, disable the successful tasks manually, and run the workflow again.
- D
Modify the workflow’s settings to automatically retry failed tasks and wait for the next scheduled run.
Show answer and explanation
Correct answer: A
Explanation
Databricks workflows provide a 'Repair and Rerun' option to efficiently address failed tasks in a workflow. This feature allows users to rerun only the failed tasks without re-executing tasks that have already succeeded, saving time and resources. The other options involve inefficient or error-prone methods that do not leverage the built-in workflow management features of Databricks.
- A. Correct.
This is the correct approach as Databricks provides a 'Repair and Rerun' option in the workflow UI that allows you to rerun only the failed tasks without re-executing successful ones.
- B. Incorrect.
This is not efficient because restarting the entire workflow would re-execute all tasks, including those that succeeded, which wastes resources and time.
- C. Incorrect.
Cloning and manually disabling tasks is unnecessary and error-prone when Databricks provides a built-in mechanism for repairing and rerunning failed tasks.
- D. Incorrect.
While automatic retries are useful, they occur during the initial run of a task. This option does not address repairing and rerunning a failed task in a workflow after the initial run has completed.