Databricks Machine Learning Professional Question 171
Select 2A data engineering team is building an ETL pipeline to process real-time sensor data from IoT devices. They need to ensure low-latency processing, handle late-arriving data, and write the results to a Delta table for downstream analytics. Why is Databricks Structured Streaming an appropriate tool for this use case?
- A
It supports stateful processing, allowing the system to handle late-arriving data and update results incrementally.
- B
It can only process batch data sources, which ensures better consistency for real-time data.
- C
It integrates seamlessly with Delta Lake, ensuring ACID compliance and scalable storage for real-time data.
- D
It requires manual implementation of fault tolerance, making it less suitable for real-time ETL pipelines.
Show answer and explanation
Correct answers: A, C
Explanation
Databricks Structured Streaming is a powerful tool for real-time ETL pipelines because it supports stateful processing for handling late-arriving data and integrates with Delta Lake to ensure durability, scalability, and ACID compliance. These capabilities make it an optimal choice for processing real-time sensor data in a low-latency, fault-tolerant manner.
- A. Correct.
Correct. Structured Streaming supports stateful processing, allowing it to manage late-arriving events and update intermediate results incrementally, which is crucial for real-time ETL pipelines.
- B. Incorrect.
Incorrect. Structured Streaming is specifically designed for real-time data processing and supports both streaming and batch data sources, making it a versatile tool for ETL pipelines.
- C. Correct.
Correct. Structured Streaming integrates with Delta Lake, enabling ACID compliance, scalable storage, and efficient append-only writes, which are vital for building robust ETL pipelines.
- D. Incorrect.
Incorrect. Structured Streaming provides built-in fault tolerance through checkpointing and retries, which makes it well-suited for real-time ETL pipelines without requiring manual fault tolerance implementation.