Databricks Data Engineer Professional Question 247
Select 3You are tasked with deploying a structured streaming job in Databricks that reads data from a Kafka source and writes the output to a Delta table. To ensure the job is monitored and operates reliably in production, which of the following steps should you take?
- A
Use checkpointing to maintain state and enable recovery in case of failures.
- B
Set the trigger interval to 'processingTime=0' to process data as quickly as possible.
- C
Enable Auto Scaling for the cluster running the streaming job to handle variable workloads.
- D
Write the output in Append mode to ensure data is not overwritten or duplicated.
- E
Use the Databricks REST API to set up alerts for job failures or performance issues.
Show answer and explanation
Correct answers: A, C, E
Explanation
A reliable and well-monitored streaming job in Databricks requires checkpointing for fault tolerance, Auto Scaling for handling variable workloads, and proactive monitoring using alerts. While the trigger interval and write mode are important considerations, they are not directly related to ensuring reliability and monitoring in this scenario.
- A. Correct.
Checkpointing is crucial in structured streaming to maintain state and allow recovery from failures, making it essential for reliable operations.
- B. Incorrect.
Setting the trigger interval to 'processingTime=0' can cause excessive resource utilization and is not recommended for production streaming jobs.
- C. Correct.
Enabling Auto Scaling ensures that the cluster can handle variable workloads efficiently, which is critical for streaming jobs with fluctuating data volumes.
- D. Incorrect.
Writing in Append mode is not always necessary or suitable for all use cases. The choice of write mode depends on the specific requirements of the job and does not inherently improve monitoring or reliability.
- E. Correct.
Using the Databricks REST API to configure alerts helps you proactively monitor job failures and performance issues, improving the overall reliability of the streaming job.