Databricks Data Engineer Professional Question 252
Select 3You are tasked with setting up a streaming job in Databricks to process real-time data from a Kafka topic and save the results to a Delta table. After deploying the job, you need to monitor its performance and ensure fault tolerance. Which of the following actions should you take to achieve this?
- A
Enable checkpointing in the streaming query.
- B
Monitor the streaming job using the Databricks Spark UI.
- C
Manually restart the job every hour to prevent memory leaks.
- D
Configure a Delta table as the sink to ensure exactly-once semantics.
- E
Use the Autoloader feature to automatically manage schema inference.
Show answer and explanation
Correct answers: A, B, D
Explanation
To deploy and monitor streaming jobs in Databricks effectively, it is essential to use checkpointing for fault tolerance, monitor the job's performance through the Spark UI, and configure a Delta table as the sink to ensure data consistency. Manually restarting jobs is not necessary, and Autoloader, while useful for batch ingestion, does not directly address the requirements of this scenario.
- A. Correct.
Enabling checkpointing is critical for fault tolerance in streaming jobs, as it allows the system to recover from failures by storing the progress of the stream.
- B. Correct.
The Databricks Spark UI provides a detailed view of the streaming job's performance, including metrics such as batch duration and processing rates.
- C. Incorrect.
Manually restarting the job is not a recommended practice, as Databricks streaming jobs are designed to run continuously. Memory leaks should be addressed through proper job optimization and memory management.
- D. Correct.
Configuring a Delta table as the sink ensures exactly-once semantics, which is essential for achieving data consistency in streaming jobs.
- E. Incorrect.
While Autoloader is a powerful tool for ingesting data in a schema-aware manner, it is unrelated to monitoring or fault tolerance of streaming jobs.