Databricks Data Engineer Associate Question 374
Select 3You are designing a data pipeline in Databricks, where the data must be processed with minimal latency to meet near real-time requirements. However, you also need to consider the cost implications of the pipeline. Which of the following statements correctly compare triggered and continuous pipelines in terms of cost and latency?
- A
Triggered pipelines generally incur lower costs as they process data in micro-batches, which allows for more efficient resource utilization.
- B
Continuous pipelines are better suited for achieving low-latency processing as they process data in real-time without waiting for batches to form.
- C
Triggered pipelines are ideal for scenarios requiring the lowest possible latency as they process data as soon as it arrives.
- D
Continuous pipelines typically incur higher costs as they maintain streaming operations and require more compute resources.
- E
Both triggered and continuous pipelines provide the same latency but differ significantly in cost based on the workload.
Show answer and explanation
Correct answers: A, B, D
Explanation
Triggered pipelines process data in micro-batches, making them cost-efficient but introducing some latency. Continuous pipelines, on the other hand, process data in real-time, achieving low latency but at the expense of higher compute costs. The choice between the two depends on the specific requirements of the workload, such as the need for real-time processing versus cost optimization.
- A. Correct.
Triggered pipelines process data in micro-batches, which can be scheduled and optimized for resource usage, leading to lower costs compared to continuous pipelines.
- B. Correct.
Continuous pipelines process data as it arrives in real-time, making them suitable for scenarios where low latency is critical.
- C. Incorrect.
Triggered pipelines do not provide the lowest possible latency because they rely on micro-batches, which introduce some delay.
- D. Correct.
Continuous pipelines maintain active streaming operations, which generally require more compute resources, leading to higher costs compared to triggered pipelines.
- E. Incorrect.
Both triggered and continuous pipelines differ in latency; continuous pipelines are optimized for low latency, while triggered pipelines focus on cost efficiency.