Databricks Data Engineer Professional Question 291
Select 2You are designing a production streaming job in Databricks that processes real-time IoT sensor data. The job has strict latency SLAs of 5 seconds for processing each batch, and your team is also tasked with minimizing cloud infrastructure costs. Which of the following design decisions will best help you meet both the latency and cost requirements?
- A
Use Auto-scaling clusters to dynamically allocate resources based on workload demand.
- B
Increase the micro-batch interval to 10 seconds to reduce processing overhead.
- C
Use Delta Lake's optimized writes to avoid excessive file creation during streaming.
- D
Enable checkpointing to ensure exactly-once processing in case of job failures.
- E
Set a fixed cluster size with maximum nodes to ensure low latency at all times.
Show answer and explanation
Correct answers: A, C
Explanation
To meet both cost and latency SLAs, you should prioritize solutions that dynamically adapt compute resources to workload demand and improve the efficiency of data processing. Auto-scaling clusters help balance cost and performance by allocating resources as needed, while Delta Lake's optimized writes reduce unnecessary overhead, ensuring both cost and latency requirements are met.
- A. Correct.
Using Auto-scaling clusters allows Databricks to dynamically adjust resources based on workload demand, which helps control cloud costs while ensuring adequate resources are available to meet latency requirements.
- B. Incorrect.
Increasing the micro-batch interval to 10 seconds may reduce processing overhead but will fail to meet the 5-second latency SLA, as this approach increases latency.
- C. Correct.
Using Delta Lake's optimized writes minimizes excessive file creation, which reduces compute overhead and storage costs, making it a cost-efficient choice that does not compromise latency.
- D. Incorrect.
While checkpointing ensures exactly-once processing, it does not directly impact cost control or latency reduction. It is more relevant to ensuring data consistency in the job.
- E. Incorrect.
Setting a fixed cluster size with maximum nodes may guarantee low latency, but it will lead to unnecessarily high infrastructure costs even during low workload periods.