Databricks Data Engineer Professional Question 289
Select 2A data engineering team is tasked with designing a production streaming system in Databricks for real-time data ingestion and processing. The system must meet the following requirements:
- Ensure latency does not exceed 5 seconds for downstream consumers.
- Minimize costs by optimizing cluster usage.
- Handle variations in streaming data volume efficiently.
Which of the following actions should the team take to meet these requirements?
- A
Use Auto-Scaling clusters to dynamically adjust cluster resources based on streaming workload.
- B
Set the trigger interval in Structured Streaming to 'ProcessingTime(5 seconds)' to ensure low-latency processing.
- C
Persist intermediate results to cloud storage after every micro-batch to reduce cluster memory usage.
- D
Use Delta Lake's Optimize and Z-Order features on streaming output to improve query performance.
- E
Configure the streaming job to run in a high-concurrency cluster to handle spikes in data volume.
Show answer and explanation
Correct answers: A, B
Explanation
To meet the requirements of low latency and cost optimization, the team should focus on leveraging Auto-scaling clusters to manage resources efficiently and setting an appropriate trigger interval to process data frequently. These actions ensure the system maintains low-latency processing while minimizing resource costs. Persisting intermediate results and using high-concurrency clusters do not specifically address the latency or cost concerns for streaming jobs.
- A. Correct.
This is correct. Auto-scaling clusters dynamically adjust resources to match the workload, optimizing costs by using only the required resources during low-demand periods while scaling up during spikes.
- B. Correct.
This is correct. Setting the trigger interval to a low value like 'ProcessingTime(5 seconds)' ensures that the system processes data frequently, reducing latency and meeting the SLA.
- C. Incorrect.
This is incorrect. Persisting intermediate results after every micro-batch can lead to increased storage costs and might not significantly improve performance or memory usage in a well-optimized streaming system.
- D. Incorrect.
This is incorrect. While Delta Lake's Optimize and Z-Order features are useful for querying batch data, they do not directly address real-time streaming latency or cost control.
- E. Incorrect.
This is incorrect. High-concurrency clusters are designed for multiple concurrent users and workloads, but they are not inherently optimized for cost or latency in streaming jobs.