Databricks Data Engineer Professional Question 293
Select 2A team is designing a production streaming job in Databricks to process real-time IoT sensor data. The system must meet strict latency SLAs of under 5 seconds while minimizing cloud costs. Which of the following design decisions would help the team optimize for both cost and latency in this scenario?
- A
Use Autopilot mode for cluster scaling to ensure resources are provisioned dynamically based on load.
- B
Increase the batch interval in Structured Streaming to process larger chunks of data less frequently.
- C
Leverage Delta Lake's optimized writes and file compaction to reduce storage costs and improve query performance.
- D
Use Delta Live Tables with a 'Complete' output mode to minimize processing delays.
- E
Configure the streaming job to use a smaller number of partitions for the input data source.
Show answer and explanation
Correct answers: A, C
Explanation
To meet strict latency SLAs while minimizing costs, leveraging cluster autoscaling ensures resources are dynamically allocated as needed, and using Delta Lake optimizations reduces costs and improves performance. Other options either increase latency or reduce parallelism, making them unsuitable for this scenario.
- A. Correct.
Autopilot mode for cluster scaling dynamically adjusts resources based on workload, ensuring cost efficiency while maintaining the necessary resources to meet latency SLAs.
- B. Incorrect.
Increasing the batch interval would increase latency, violating the SLA requirement of under 5 seconds.
- C. Correct.
Delta Lake's optimized writes and file compaction improve both cost efficiency and performance by reducing the overhead associated with small files and optimizing query execution.
- D. Incorrect.
Using 'Complete' output mode in Delta Live Tables processes the entire dataset with each update, which could introduce significant latency for large datasets.
- E. Incorrect.
Configuring a smaller number of partitions would reduce parallelism, potentially increasing processing time and violating the latency SLA.