Databricks Data Engineer Professional Question 251
Select 3You are managing a Databricks workspace where a streaming job is processing real-time data from a Kafka source and writing it to a Delta table. The job has been running successfully for weeks, but recently it started failing intermittently due to checkpoint corruption. Which of the following actions should you take to resolve the issue and ensure the job runs reliably in the future?
- A
Delete the corrupted checkpoint directory and restart the streaming job from the latest available offset.
- B
Enable the 'ignoreCorruptFiles' option in the streaming query to bypass corrupted checkpoint files.
- C
Configure the streaming job with a new checkpoint directory and reset the offsets to 'earliest'.
- D
Set up monitoring using Databricks SQL queries on the Delta table to track data ingestion issues.
- E
Implement a retry mechanism in the job to handle intermittent failures gracefully.
Show answer and explanation
Correct answers: A, C, E
Explanation
Checkpoint corruption can disrupt the reliability of a streaming job in Databricks. Deleting the corrupted checkpoint directory or configuring a new checkpoint directory allows the streaming job to resume or restart without encountering the corruption. Implementing a retry mechanism helps the job handle transient issues, ensuring overall reliability. While monitoring tools like Databricks SQL can provide insights into ingestion issues, they do not directly address checkpoint corruption.
- A. Correct.
Deleting the corrupted checkpoint directory and restarting the job ensures that the corruption is resolved. However, this will cause the job to reprocess data starting from the latest available offset, which may lead to some data duplication.
- B. Incorrect.
The 'ignoreCorruptFiles' option is not a valid configuration for resolving corrupted checkpoints in streaming jobs. This is typically used for handling corrupt data files in batch jobs.
- C. Correct.
Configuring a new checkpoint directory and resetting the offsets to 'earliest' ensures that the streaming job starts fresh, avoiding the corrupted checkpoint files. However, this may lead to reprocessing all historical data.
- D. Incorrect.
Setting up monitoring with Databricks SQL queries is useful for tracking ingestion issues but does not directly resolve checkpoint corruption in streaming jobs.
- E. Correct.
A retry mechanism ensures that intermittent failures are handled gracefully, improving the job's reliability and reducing downtime.