Databricks Data Engineer Professional Question 274
Select 2A Databricks job that processes daily sales data has failed due to a transient network issue while writing data to an external storage system. The job consists of three tasks, and the failure occurred during the second task. As a data engineer, you need to repair the job and ensure that it resumes processing efficiently from the point of failure. What steps should you take to repair and rerun the job?
- A
Use the Databricks job repair functionality to rerun only the failed task and any dependent tasks.
- B
Manually delete the output of all previous tasks and rerun the entire job from the beginning.
- C
Check the job run details and logs to identify the specific cause of failure before using the repair functionality.
- D
Modify the job configuration to skip the failed task and manually reprocess its output later.
- E
Restart the cluster associated with the job to ensure that no residual issues affect the rerun.
Show answer and explanation
Correct answers: A, C
Explanation
To repair a failed job in Databricks, it is best practice to use the job repair functionality, which enables efficient reruns of only the failed and dependent tasks. Additionally, reviewing logs and job details ensures the root cause of the failure is addressed before rerunning the job. This approach minimizes reprocessing time and ensures data correctness.
- A. Correct.
This is correct. Databricks provides a job repair functionality that allows you to rerun only the failed tasks and their dependent tasks, reducing unnecessary reprocessing.
- B. Incorrect.
This is incorrect. Manually deleting outputs and rerunning the entire job is inefficient and goes against best practices for handling failed jobs in Databricks.
- C. Correct.
This is correct. Reviewing the job run details and logs helps you understand why the task failed, ensuring the issue is resolved before using the repair functionality.
- D. Incorrect.
This is incorrect. Skipping the failed task and reprocessing its output later introduces manual overhead and risks inconsistent data.
- E. Incorrect.
This is incorrect. Restarting the cluster is not necessary unless the failure was caused by a cluster-specific issue, which is not indicated in this scenario.