Databricks Machine Learning Professional Question 170
Select 3You are building a real-time ETL pipeline to process streaming sensor data from IoT devices and store the results in a Delta Lake table for downstream machine learning tasks. Which of the following describes why Structured Streaming in Databricks is a suitable choice for this use case?
- A
Structured Streaming provides exactly-once processing guarantees, ensuring the data is processed reliably.
- B
It allows for seamless integration with Delta Lake, enabling efficient incremental updates to the target table.
- C
Structured Streaming requires all input data to be fully loaded into memory, which makes it faster than other streaming frameworks.
- D
It supports both batch and streaming workloads using the same APIs, making the ETL pipeline easier to maintain.
- E
Structured Streaming automatically handles schema evolution in Delta tables without requiring additional configuration.
Show answer and explanation
Correct answers: A, B, D
Explanation
Structured Streaming is a powerful tool for building real-time ETL pipelines because it ensures reliable processing with exactly-once guarantees, integrates effortlessly with Delta Lake for incremental updates, and provides unified APIs for batch and streaming workloads. These features make it an excellent choice for processing streaming sensor data in a scalable and maintainable way.
- A. Correct.
Structured Streaming provides exactly-once processing guarantees by leveraging checkpointing, ensuring reliable and consistent data processing. This is critical for real-time ETL pipelines.
- B. Correct.
Structured Streaming integrates seamlessly with Delta Lake, enabling efficient incremental updates to the target table, which is ideal for streaming data ingestion.
- C. Incorrect.
This is incorrect. Structured Streaming does not require all input data to be loaded into memory. In fact, it processes data incrementally in micro-batches or continuous processing mode, making it memory-efficient.
- D. Correct.
Structured Streaming supports the same APIs for batch and streaming workloads. This simplifies the development and maintenance of ETL pipelines as the same code can handle both types of workloads.
- E. Incorrect.
While Delta Lake supports schema evolution, Structured Streaming does not automatically handle such changes without user intervention. This option is incorrect.