Databricks Data Engineer Professional Question 246
Select 3You are designing a production streaming job in Databricks to process real-time data from IoT sensors. The job has stringent latency SLAs of under 5 seconds for processing and a requirement to minimize costs. Which of the following design decisions would help you achieve these goals?
- A
Use Auto Scaling for the cluster to dynamically adjust resources based on workload demands.
- B
Increase the batch interval for micro-batches to 10 seconds to reduce processing overhead.
- C
Leverage Delta Live Tables (DLT) with partition pruning to optimize data reads.
- D
Persist intermediate results in a high-cost storage format such as JSON for better compatibility.
- E
Use a Structured Streaming checkpoint location in a low-latency storage like DBFS.
Show answer and explanation
Correct answers: A, C, E
Explanation
To design a cost-efficient and low-latency streaming job, you can leverage Auto Scaling to optimize resource usage, use Delta Live Tables with partition pruning to improve data read performance, and ensure low-latency checkpointing in DBFS to support quick state recovery. These steps collectively address both cost and latency considerations, ensuring that the streaming job meets the SLA requirements.
- A. Correct.
Auto Scaling allows the cluster to dynamically scale up or down based on workload demands, ensuring optimal resource usage and cost efficiency while meeting latency requirements.
- B. Incorrect.
Increasing the batch interval to 10 seconds can reduce processing overhead but will violate the 5-second latency SLA, making it an unsuitable choice.
- C. Correct.
Delta Live Tables (DLT) with partition pruning optimizes data reads, speeding up processing and helping to meet the 5-second latency SLA effectively.
- D. Incorrect.
Persisting intermediate results in a high-cost format like JSON is inefficient and increases costs unnecessarily. It does not align with the goal of minimizing costs.
- E. Correct.
Using a low-latency storage like DBFS for the Structured Streaming checkpoint location ensures minimal delays in state recovery and progress tracking, helping to meet latency SLAs.