Databricks Data Engineer Professional Question 294
Select 3You are tasked with designing a production streaming job in Databricks that processes real-time data from a Kafka source. The system must meet the following requirements:
- Maintain a low processing latency SLA of under 1 second.
- Keep operational costs within the budget by minimizing unnecessary resource usage. Which of the following strategies should you implement to meet both latency and cost requirements?
- A
Use Auto Scaling for the cluster to dynamically adjust resources based on workload.
- B
Configure the streaming job with a trigger interval of 5 seconds to reduce resource utilization.
- C
Use Delta Lake for checkpointing to ensure fault tolerance and efficient recovery during failures.
- D
Set a fixed high number of streaming partitions to maximize parallelism at all times.
- E
Enable optimized writes and auto-compaction in Delta Lake to reduce downstream costs.
Show answer and explanation
Correct answers: A, C, E
Explanation
To design a cost-effective and low-latency streaming job in Databricks, it is essential to strike a balance between resource allocation and performance. Auto Scaling helps minimize costs by adjusting resources based on workload, while Delta Lake checkpointing ensures fault tolerance and recovery to maintain low latency. Enabling optimized writes and auto-compaction further reduces downstream costs without compromising processing efficiency.
- A. Correct.
Auto Scaling dynamically adjusts the cluster resources based on streaming workload, helping to control costs while maintaining low latency during peak loads.
- B. Incorrect.
Configuring a trigger interval of 5 seconds increases latency and may not meet the sub-1-second SLA requirement, even though it reduces resource utilization.
- C. Correct.
Delta Lake checkpointing ensures fault tolerance, which is crucial for maintaining consistent low-latency processing in production environments.
- D. Incorrect.
Setting a fixed high number of partitions may unnecessarily increase resource usage and costs when the workload is low, making it less cost-efficient.
- E. Correct.
Optimized writes and auto-compaction in Delta Lake reduce the cost of downstream operations, such as querying and storage, while maintaining efficiency.