Databricks Data Engineer Professional Question 243
Select 3You are designing a production streaming job in Databricks that processes real-time IoT sensor data from devices. The job has strict SLA requirements to process data within 2 seconds, and cost optimization is a priority. Which of the following configurations and strategies would best help you meet both the SLA and cost requirements?
- A
Use Auto Scaling for the cluster to dynamically adjust resources based on workload demand.
- B
Set a high trigger interval (e.g., 60 seconds) for the Structured Streaming query to reduce resource usage.
- C
Use Delta Lake for streaming sinks to enable efficient upserts and minimize storage costs.
- D
Enable Adaptive Query Execution (AQE) to optimize query plans dynamically during execution.
- E
Configure a low micro-batch interval (e.g., 1 second) to ensure data is processed quickly.
Show answer and explanation
Correct answers: A, C, E
Explanation
To meet both cost and latency SLAs for a production streaming job, you need to balance resource efficiency and processing speed. Auto Scaling ensures the cluster dynamically adjusts to workload demands, optimizing costs. Delta Lake as the streaming sink minimizes storage overhead and enables efficient data handling. Finally, setting a low micro-batch interval ensures timely processing of data, meeting the strict latency SLA. Avoid high trigger intervals as they increase latency, and AQE is less relevant for streaming scenarios.
- A. Correct.
Auto Scaling allows the cluster to dynamically adjust resources based on the workload, which helps in maintaining cost efficiency while ensuring enough resources are available to meet latency SLAs.
- B. Incorrect.
Setting a high trigger interval (e.g., 60 seconds) increases latency as the job will only process data in larger intervals, making it unsuitable for low-latency requirements.
- C. Correct.
Using Delta Lake as the streaming sink ensures efficient storage and upsert capabilities, which reduces both storage costs and processing overhead.
- D. Incorrect.
Adaptive Query Execution (AQE) is primarily designed for batch processing optimizations and does not directly benefit latency or cost control in streaming jobs.
- E. Correct.
Configuring a low micro-batch interval ensures faster data processing, which aligns with the low-latency SLA of 2 seconds.