Databricks Machine Learning Professional Question 168
Select 3A data engineering team is building an ETL pipeline to process real-time sensor data from IoT devices and load it into a Delta Lake table for downstream machine learning tasks. The team decides to use Structured Streaming in Databricks. Which of the following reasons justify using Structured Streaming for this use case?
- A
Structured Streaming provides incremental processing capabilities, allowing the pipeline to process data as it arrives in micro-batches or in continuous mode.
- B
Structured Streaming automatically handles late-arriving data and supports watermarking to define event-time windows.
- C
Structured Streaming can only process batch data and is unsuitable for real-time ETL pipelines.
- D
Structured Streaming enables schema enforcement and evolution, ensuring the data is consistently formatted in the Delta Lake table.
- E
Structured Streaming is designed for real-time streaming but requires manual checkpointing for fault tolerance.
Show answer and explanation
Correct answers: A, B, D
Explanation
Structured Streaming is a powerful tool for building real-time ETL pipelines due to its ability to process data incrementally, handle late-arriving data with watermarking, and enforce schema consistency in Delta Lake integrations. These features make it ideal for scenarios like processing IoT sensor data in real-time. Incorrect options either misrepresent Structured Streaming's capabilities or describe features it does not require, such as manual checkpointing.
- A. Correct.
Correct. Structured Streaming processes data incrementally, which is essential for real-time ETL pipelines where data is constantly arriving. This makes it suitable for IoT data processing.
- B. Correct.
Correct. Structured Streaming supports late-arriving data and watermarking, which are critical for handling real-world streaming scenarios with event-time windows.
- C. Incorrect.
Incorrect. Structured Streaming is specifically designed for both real-time and micro-batch processing, making it ideal for real-time ETL pipelines.
- D. Correct.
Correct. Structured Streaming integrates with Delta Lake and supports schema enforcement and evolution, ensuring that the data ingested into the table adheres to a consistent format.
- E. Incorrect.
Incorrect. Structured Streaming includes automatic checkpointing to ensure fault tolerance; manual checkpointing is not required.