Databricks Data Engineer Professional Question 273
Select 2A data engineering team is running a Databricks job that processes streaming data and writes it to a Delta Lake table. The job failed midway due to a transient networking issue. To recover and successfully complete the job, which actions should the team take?
- A
Review the job run details to identify the failure point and restart the job from the last successful checkpoint.
- B
Delete the Delta Lake table to ensure the next run starts with a clean slate.
- C
Enable the 'Repair and rerun' option in the job configuration to automatically handle partial writes in Delta Lake.
- D
Manually remove the partially processed files in the Delta Lake table directory before restarting the job.
- E
Use the Delta Lake transaction logs to ensure idempotency and restart the job from the last checkpoint.
Show answer and explanation
Correct answers: A, E
Explanation
To recover from a failed Databricks job, the team should leverage Delta Lake's transaction logs and the checkpointing mechanism provided by structured streaming. This ensures that the job can resume from the last successful state without data loss or duplication. Deleting the table or manually managing files introduces risks and is not aligned with best practices.
- A. Correct.
Correct: Reviewing the job run details and restarting from the last successful checkpoint is essential to recovering from a transient failure in a streaming job.
- B. Incorrect.
Incorrect: Deleting the Delta Lake table is unnecessary and can lead to loss of data that has already been processed and written successfully.
- C. Incorrect.
Incorrect: There is no 'Repair and rerun' option in Databricks job configurations. Recovery is typically handled through checkpoints and Delta Lake features.
- D. Incorrect.
Incorrect: Manually removing partially processed files is error-prone and unnecessary since Delta Lake maintains transaction logs and ensures consistency.
- E. Correct.
Correct: Delta Lake transaction logs track all changes and enable idempotent operations, which ensures that rerunning the job will not reprocess data that has already been successfully written.