Databricks Data Engineer Associate Question 376
Select 3You are designing a data pipeline in Databricks for a streaming application. The pipeline must minimize latency to deliver near real-time results, but cost optimization is also important. Which of the following statements accurately compare triggered and continuous pipelines in this context?
- A
Triggered pipelines process data in micro-batches, which can result in higher latency compared to continuous pipelines.
- B
Continuous pipelines provide lower latency by processing data record-by-record, but this can increase the operational cost for long-running processes.
- C
Triggered pipelines are generally more cost-efficient for workloads that do not require extremely low latency.
- D
Continuous pipelines are better suited for scenarios where both low latency and minimal cost are required, compared to triggered pipelines.
Show answer and explanation
Correct answers: A, B, C
Explanation
Triggered and continuous pipelines differ significantly in how they handle streaming data. Triggered pipelines process data in micro-batches, resulting in higher latency but lower costs, making them suitable for use cases that do not demand real-time processing. Continuous pipelines are designed for record-by-record processing, minimizing latency at the cost of higher resource utilization. Understanding the trade-offs between latency and cost is critical when selecting the appropriate pipeline type for a use case.
- A. Correct.
Triggered pipelines use micro-batch processing, which introduces additional latency as data is processed in discrete chunks rather than continuously. This is correct.
- B. Correct.
Continuous pipelines process data record-by-record, which reduces latency but may increase costs due to the need for constant resource usage. This is correct.
- C. Correct.
Triggered pipelines are more cost-efficient for workloads that do not demand extremely low latency because they do not require resources to be constantly active. This is correct.
- D. Incorrect.
This option is incorrect because continuous pipelines prioritize low latency over cost-efficiency and are generally more expensive than triggered pipelines for long-running tasks.