Databricks Data Engineer Professional Question 241
Select 3A data engineering team manages a production streaming job in Databricks that processes real-time clickstream data. The job must adhere to a Service Level Agreement (SLA) that requires both low latency (processing data within 5 seconds) and cost efficiency. Which of the following actions should the team take to design a system that meets these requirements?
- A
Optimize the streaming job by using Trigger.Once to process the data in batches to reduce costs.
- B
Enable auto-scaling for the cluster to efficiently handle variable data loads while controlling costs.
- C
Use Delta Lake to checkpoint progress and ensure exactly-once processing semantics.
- D
Minimize the watermark delay in the streaming query to reduce latency for late-arriving data.
- E
Configure the streaming query to use a high number of partitions to parallelize processing, regardless of the input data volume.
Show answer and explanation
Correct answers: B, C, D
Explanation
To meet both the cost and latency SLAs for a production streaming job, the team must balance performance optimizations with cost efficiency. Auto-scaling ensures the cluster uses resources dynamically to match the workload, Delta Lake ensures efficient processing with exactly-once semantics, and minimizing the watermark delay reduces latency for late-arriving events. Together, these actions help meet the SLA requirements for both cost and latency.
- A. Incorrect.
Trigger.Once processes data in discrete batches and is not suitable for low-latency requirements in real-time streaming jobs. This approach sacrifices latency for cost savings, which violates the SLA.
- B. Correct.
Enabling auto-scaling allows the cluster to dynamically adjust resources based on the workload, helping to control costs while maintaining the required performance.
- C. Correct.
Delta Lake provides checkpointing and exactly-once processing, which are essential for stable and efficient streaming job design. This helps avoid reprocessing data unnecessarily, improving both cost and reliability.
- D. Correct.
Minimizing the watermark delay ensures that late-arriving data is processed faster, reducing end-to-end latency and meeting the low-latency SLA.
- E. Incorrect.
Using a high number of partitions without considering input data volume can lead to resource overprovisioning and increased costs. This approach does not align with the requirement for cost efficiency.