Databricks Data Engineer Professional Question 248
Select 3You are tasked with deploying a structured streaming job in Databricks that processes real-time IoT sensor data and writes the output to a Delta table. To ensure the job runs reliably, you must implement monitoring and alerting for failed streaming queries. Which of the following steps should you perform to achieve this?
- A
Use Databricks Jobs to schedule the streaming query and configure retry logic.
- B
Set up a streaming query listener to collect metrics and log them to a monitoring system.
- C
Enable Delta Lake checkpointing to ensure exactly-once processing.
- D
Configure Databricks SQL Alerts to monitor the Delta table for anomalies in the data.
- E
Manually restart the streaming job each time it fails.
Show answer and explanation
Correct answers: A, B, C
Explanation
Deploying and monitoring a streaming job in Databricks requires a combination of tools and techniques to ensure reliability and observability. Databricks Jobs can handle job orchestration and retries, streaming query listeners provide critical metrics for monitoring, and Delta Lake checkpointing ensures fault tolerance. While Databricks SQL Alerts can help monitor data anomalies, they are not sufficient for managing streaming query failures. Manual interventions are also not a recommended approach for production systems.
- A. Correct.
Correct: Databricks Jobs can manage the lifecycle of streaming queries, including retrying failed jobs to improve reliability.
- B. Correct.
Correct: Streaming query listeners allow you to gather metrics like throughput, latency, and failure rates, which are essential for monitoring and alerting.
- C. Correct.
Correct: Delta Lake checkpointing ensures fault tolerance and exactly-once processing, which is critical for reliable streaming jobs.
- D. Incorrect.
Incorrect: While Databricks SQL Alerts are useful for monitoring data, they are not directly related to the reliability or monitoring of the streaming query itself.
- E. Incorrect.
Incorrect: Manually restarting the job each time it fails is not a scalable or reliable solution. Automated retry mechanisms should be used instead.