Databricks Data Engineer Professional Question 244
Select 3You are designing a production streaming job in Databricks to process real-time data with strict latency SLAs and cost constraints. The job ingests data from a Kafka topic, applies transformations, and writes the results to Delta Lake. Which of the following design choices would help you optimize for both cost and latency?
- A
Use Auto Scaling for your Databricks cluster to dynamically adjust resources based on workload.
- B
Set the trigger interval for the streaming job to 'Trigger.Once' to minimize compute costs.
- C
Enable Delta Lake's optimized writes and Z-Ordering to improve downstream query performance.
- D
Use a smaller cluster with fewer workers to minimize initial cluster costs.
- E
Implement Structured Streaming with a micro-batch mode and a trigger interval based on SLA requirements.
Show answer and explanation
Correct answers: A, C, E
Explanation
Optimizing for cost and latency in production streaming jobs involves balancing resource allocation with workload demands. Auto Scaling ensures efficient resource usage, Delta Lake's optimization features improve downstream efficiency, and using micro-batch mode with a well-configured trigger interval aligns processing with latency SLAs. These design choices collectively address both cost and latency constraints effectively.
- A. Correct.
Auto Scaling helps control costs by dynamically adjusting cluster resources to match the workload, ensuring you don't over-allocate resources during periods of low usage while still meeting latency requirements during peak times.
- B. Incorrect.
'Trigger.Once' processes all the available data in a single batch and then stops the stream, which is not suitable for real-time streaming jobs with strict latency SLAs. This option would increase latency and is not a viable solution.
- C. Correct.
Delta Lake's optimized writes and Z-Ordering improve downstream query performance and reduce the overhead of subsequent read operations, indirectly supporting cost and latency optimizations.
- D. Incorrect.
Using a smaller cluster with fewer workers might reduce initial costs but can lead to higher latency and potential SLA violations, making this choice counterproductive.
- E. Correct.
Structured Streaming with micro-batch mode and a trigger interval aligned to SLA requirements ensures predictable latency while leveraging Databricks' efficient resource management for cost control.