Databricks Data Engineer Associate Question 373
Select 3A data engineering team is tasked with designing a Delta Live Table (DLT) pipeline for processing real-time data from IoT sensors. The team needs to decide whether to use a triggered or continuous pipeline. Which of the following statements correctly compare triggered and continuous pipelines in terms of cost and latency?
- A
Triggered pipelines can result in lower overall costs because they process data in scheduled or manual intervals rather than continuously consuming resources.
- B
Continuous pipelines are generally better for low-latency requirements as they process data in near real-time.
- C
Triggered pipelines are better suited for use cases requiring low-latency processing of streaming data.
- D
Continuous pipelines can lead to higher costs due to the persistent consumption of resources for real-time processing.
- E
Triggered pipelines are ideal for scenarios with high data volume and strict low-latency requirements.
Show answer and explanation
Correct answers: A, B, D
Explanation
Triggered pipelines are designed for batch-style processing, typically activated on a schedule or manually, which results in lower costs but higher latency. Continuous pipelines, on the other hand, process data in real-time, making them better suited for low-latency use cases but at the expense of higher costs due to persistent resource utilization. Understanding the trade-offs between cost and latency is crucial when designing pipelines for specific use cases.
- A. Correct.
Correct: Triggered pipelines process data in scheduled, batch-like intervals, which can reduce costs since resources are only consumed during the triggers.
- B. Correct.
Correct: Continuous pipelines process data in near real-time, making them suitable for low-latency use cases where delays must be minimized.
- C. Incorrect.
Incorrect: Triggered pipelines are not designed for low-latency streaming; they are more suited for batch or scheduled workloads with tolerable delays.
- D. Correct.
Correct: Continuous pipelines consume resources persistently for real-time processing, which can increase costs compared to triggered pipelines.
- E. Incorrect.
Incorrect: Triggered pipelines are not ideal for scenarios requiring both high data volume and strict low-latency requirements. Continuous pipelines are better suited for such use cases.