Databricks Data Engineer Professional Question 242
Select 2You are tasked with designing a production streaming job in Databricks to process high-velocity data from IoT sensors. The system must meet the following requirements:
- Ensure minimal latency (under 1 second) for event processing.
- Control cloud costs by avoiding unnecessary over-provisioning.
Which of the following strategies would best meet both the cost and latency SLA requirements?
- A
Use Auto Scaling for the streaming cluster to dynamically adjust resources based on workload.
- B
Set the trigger interval to 'processingTime=1 second' and optimize for micro-batches.
- C
Use a fixed cluster size with the maximum number of nodes to ensure consistent low latency.
- D
Enable Delta Lake's Optimize Write and Optimize Z-Order features to improve downstream query performance.
- E
Leverage Structured Streaming with the 'Complete' output mode to ensure all data is recomputed in each trigger.
Show answer and explanation
Correct answers: A, B
Explanation
To meet both cost and latency SLAs for a production streaming job, it is critical to dynamically adjust resources to workload demands using Auto Scaling and to configure the system to process data at appropriate intervals (e.g., using micro-batches with a 1-second trigger). These strategies ensure minimal latency without over-provisioning resources, balancing both performance and cost. Other options either increase costs unnecessarily or fail to address real-time latency requirements.
- A. Correct.
Auto Scaling allows the cluster to dynamically adjust based on the workload, optimizing for cost while ensuring sufficient resources are available during peak loads. This is essential for balancing cost and latency requirements.
- B. Correct.
Setting the trigger interval to 'processingTime=1 second' ensures micro-batch processing occurs at a frequency that meets the latency SLA. This minimizes delay while avoiding the overhead of continuous processing.
- C. Incorrect.
Using a fixed cluster size with the maximum number of nodes may ensure low latency, but it is not cost-efficient as resources will be over-provisioned during periods of low activity.
- D. Incorrect.
Delta Lake's Optimize Write and Optimize Z-Order features improve downstream read performance, but they do not directly address latency for streaming ingestion or cost control for the streaming job.
- E. Incorrect.
The 'Complete' output mode recomputes all data in each trigger, which incurs additional latency and resource overhead, making it unsuitable for meeting the latency SLA or controlling costs.