Databricks Data Engineer Associate Question 375
Select 4You are designing a data pipeline in Databricks to process streaming data from IoT devices. The pipeline needs to provide near real-time insights while minimizing operational costs. Which of the following statements accurately compare triggered pipelines and continuous pipelines in this scenario?
- A
Triggered pipelines have higher latency compared to continuous pipelines but are typically more cost-efficient.
- B
Continuous pipelines provide lower latency than triggered pipelines but may result in higher operational costs.
- C
Triggered pipelines are more suitable for workloads where data processing needs to occur at fixed intervals.
- D
Continuous pipelines are better suited for scenarios requiring immediate processing of streaming data.
- E
Triggered pipelines are the only option for ensuring exactly-once processing in Databricks.
Show answer and explanation
Correct answers: A, B, C, D
Explanation
Triggered pipelines operate in batch mode, making them ideal for cost-sensitive use cases or periodic processing but at the expense of increased latency. Continuous pipelines process data in real time, which minimizes latency but typically incurs higher operational costs due to constant resource usage. Understanding the trade-offs between cost and latency is critical when designing pipelines for specific workloads.
- A. Correct.
Triggered pipelines do not process data continuously; they are executed at defined intervals (batch processing), leading to higher latency. However, this approach is often more cost-efficient as it avoids the constant resource consumption of continuous pipelines.
- B. Correct.
Continuous pipelines process data as it arrives, which reduces latency. However, this approach can lead to higher operational costs because it requires persistent resource usage to maintain the continuous processing.
- C. Correct.
Triggered pipelines work well for workloads with predictable, periodic processing requirements, as they execute in discrete batches.
- D. Correct.
Continuous pipelines are designed for real-time or low-latency use cases, making them ideal for scenarios where insights are needed immediately upon data arrival.
- E. Incorrect.
This statement is incorrect. Both triggered and continuous pipelines in Databricks can support exactly-once processing, depending on how the pipeline is configured, so this is not exclusive to triggered pipelines.