Databricks Data Engineer Professional Question 292
Select 3You are designing a production streaming job in Databricks that processes real-time data from IoT devices. The system must meet a Service Level Agreement (SLA) of processing each batch within 5 seconds to ensure low latency, and it should minimize cost by optimizing cluster resources. Which of the following design choices will help you meet both the latency and cost SLAs?
- A
Use Auto Scaling for the cluster to adjust the number of nodes based on workload demand.
- B
Configure the streaming job's trigger interval to match the SLA, such as 5 seconds.
- C
Use a high number of partitions for the input source to maximize parallelism, regardless of the data volume.
- D
Leverage Delta Lake for efficient stateful processing and fault tolerance.
- E
Run the streaming job on a high-cost, fixed-size cluster to ensure consistent performance.
Show answer and explanation
Correct answers: A, B, D
Explanation
To design a streaming system that meets both latency and cost SLAs, you need to balance resource efficiency with performance. Auto Scaling ensures cost control by adjusting resources dynamically. Setting the trigger interval to align with the SLA ensures that the system processes data quickly enough to meet latency requirements. Leveraging Delta Lake optimizes stateful processing and fault tolerance, further supporting the SLA adherence. Avoiding wasteful partitioning and high fixed-cost clusters aligns with cost-saving goals.
- A. Correct.
Using Auto Scaling allows the cluster to dynamically adjust resources based on workload, reducing cost during low demand while maintaining performance during peak times.
- B. Correct.
Configuring the trigger interval to match the SLA ensures that the streaming job processes data within the required latency window.
- C. Incorrect.
Using a high number of partitions regardless of data volume can lead to unnecessary overhead and inefficiencies, increasing cost without any guaranteed performance improvement.
- D. Correct.
Delta Lake provides efficient mechanisms for stateful processing, fault tolerance, and optimized I/O operations, which help in maintaining low latency while minimizing resource usage.
- E. Incorrect.
Running the job on a high-cost, fixed-size cluster may meet the latency SLA but unnecessarily increases costs, violating cost optimization requirements.