Databricks Data Engineer Professional Question 245
Select 3You are tasked with designing a streaming system on Databricks for processing real-time telemetry data. The system must meet a strict latency SLA of 5 seconds, while also minimizing costs. Which approaches should you use to ensure the system meets these requirements?
- A
Use Auto Scaling for the cluster to adjust resources based on workload demands.
- B
Increase the micro-batch interval to 10 seconds to reduce compute costs.
- C
Enable Delta Lake's Optimize Writes and Z-Order for downstream queries.
- D
Configure Structured Streaming with Trigger.Once to process data only when the pipeline is triggered.
- E
Use a smaller worker node type with more instances instead of fewer large nodes.
Show answer and explanation
Correct answers: A, C, E
Explanation
To meet both the latency SLA and cost-efficiency requirements in a streaming system, you need to balance resource allocation with workload demands. Auto Scaling ensures the cluster scales appropriately, Delta Lake optimizations reduce latency for downstream queries, and using smaller worker nodes improves parallelism and resource utilization. Increasing the micro-batch interval or using Trigger.Once would compromise the latency SLA.
- A. Correct.
Auto Scaling dynamically adjusts cluster size, helping to optimize resource usage and control costs while maintaining SLA requirements.
- B. Incorrect.
Increasing the micro-batch interval to 10 seconds violates the 5-second latency SLA, even though it may reduce costs.
- C. Correct.
Delta Lake's Optimize Writes and Z-Order improve downstream query performance by reducing storage latency and improving data layout, which indirectly supports SLA adherence.
- D. Incorrect.
Trigger.Once processes data only once when triggered, which is not suitable for a low-latency streaming system requiring continuous processing.
- E. Correct.
Using smaller worker nodes with more instances allows for better parallelism and resource utilization, which can help meet low-latency SLAs while managing costs.