Databricks Data Engineer Professional Question 290
Select 2You are designing a production streaming job on Databricks that processes real-time clickstream data from a website. The job must meet a strict SLA of processing new data within 2 seconds while minimizing cloud costs. Which of the following design decisions would help you achieve both the latency SLA and cost-efficiency?
- A
Use Auto Scaling for the cluster and set a reasonable minimum and maximum number of worker nodes.
- B
Write the streaming output to a Delta Lake table with Optimize and Z-Order executed after every batch.
- C
Use Structured Streaming's Trigger.Once mode for managing the processing intervals.
- D
Implement checkpointing to a low-cost cloud storage location to enable recovery from failure.
- E
Choose a cluster type with fewer, larger instances rather than more, smaller instances for cost efficiency.
Show answer and explanation
Correct answers: A, D
Explanation
To meet both latency SLA and cost-efficiency goals, using Auto Scaling ensures resource usage is optimized dynamically, avoiding over-provisioning or under-provisioning. Checkpointing provides fault tolerance without incurring high costs, making it a critical component for production streaming jobs. Other options either do not address the latency SLA effectively or introduce unnecessary overhead.
- A. Correct.
Using Auto Scaling allows the cluster to dynamically adjust resources based on workload, which helps balance cost and performance, especially for streaming jobs with fluctuating data rates.
- B. Incorrect.
Executing Optimize and Z-Order after every batch would add significant overhead and increase costs unnecessarily for a streaming job that requires low latency.
- C. Incorrect.
Trigger.Once processes all available data in a single batch and then stops, which is not suitable for meeting a strict latency SLA of 2 seconds in a continuous streaming environment.
- D. Correct.
Checkpointing is essential for fault tolerance and ensures that the job can resume from where it left off without reprocessing. Using low-cost cloud storage for this purpose helps control costs.
- E. Incorrect.
Choosing fewer, larger instances might reduce costs in some cases, but for streaming workloads with latency constraints, this setup could lead to underutilization of resources and higher latency, making it a less optimal choice.