Databricks Data Engineer Professional Question 249
Select 4You are tasked with deploying and monitoring a structured streaming job in Databricks that processes real-time data from a Kafka source and writes the results to a Delta table. Your organization requires the solution to be fault-tolerant and provide monitoring capabilities for data loss or lag. Which of the following steps are necessary to meet these requirements?
- A
Enable checkpointing for the streaming query.
- B
Configure a trigger interval for the streaming query.
- C
Write the processed data to a non-transactional storage format like CSV.
- D
Integrate the query with Databricks metrics to monitor progress and lag.
- E
Use the 'awaitTermination()' method to stop the streaming query automatically after processing all available data.
- F
Deploy the streaming job using a Databricks Job with a specified retry policy.
Show answer and explanation
Correct answers: A, B, D, F
Explanation
To deploy and monitor a fault-tolerant streaming job in Databricks, checkpointing is required to allow recovery of the streaming state. A trigger interval optimizes processing frequency, while Databricks metrics aid in monitoring. Deploying the job as a Databricks Job with a retry policy ensures resilience. Writing data to a transactional format (not CSV) and avoiding methods like 'awaitTermination()' further align the solution with the requirements for fault tolerance and monitoring.
- A. Correct.
Enabling checkpointing ensures fault tolerance by saving the query's state, allowing recovery in case of failure.
- B. Correct.
Configuring a trigger interval helps control the frequency at which the query processes batches of data, improving resource utilization and reducing lag.
- C. Incorrect.
Writing to a non-transactional format like CSV does not guarantee consistency or fault tolerance, making it unsuitable for this scenario.
- D. Correct.
Integrating the query with Databricks metrics provides visibility into the query's progress, lag, and other performance metrics, aiding in monitoring.
- E. Incorrect.
The 'awaitTermination()' method is not suitable for this use case as it is used to block execution until the query is manually stopped, not for fault-tolerant deployment.
- F. Correct.
Deploying the streaming job using a Databricks Job with a retry policy ensures the job can recover from transient failures during execution.