Databricks Data Engineer Professional Question 77
Single answerYou are designing a real-time data pipeline using Databricks Structured Streaming and Delta Lake. The pipeline is required to process streaming data from IoT devices, ensuring exactly-once processing semantics and providing the ability to reprocess historical data when needed. Which of the following design patterns would best meet these requirements?
- A
Using Delta Lake as a sink for the streaming data, enabling checkpointing and versioning for reprocessing
- B
Writing streaming data directly to cloud object storage without checkpointing
- C
Using in-memory storage for the streaming data to achieve faster processing
- D
Writing streaming data to Delta Lake and disabling schema enforcement to handle schema evolution automatically
Show answer and explanation
Correct answer: A
Explanation
Delta Lake is a key component for building reliable data pipelines with Structured Streaming. It ensures exactly-once processing semantics through ACID transactions and checkpointing. Additionally, Delta Lake's versioning feature allows for efficient historical data reprocessing, making it the ideal choice for this scenario.
- A. Correct.
This is the correct answer. Delta Lake enables exactly-once processing semantics and supports ACID transactions. Checkpointing ensures fault tolerance, and versioning allows for historical data reprocessing.
- B. Incorrect.
This is incorrect. Writing directly to cloud object storage without checkpointing does not guarantee exactly-once processing semantics and lacks the ability to reprocess data efficiently.
- C. Incorrect.
This is incorrect. While in-memory storage can be faster, it does not provide durability, exactly-once guarantees, or reprocessing capabilities.
- D. Incorrect.
This is incorrect. Disabling schema enforcement in Delta Lake can lead to data quality issues and does not align with best practices for handling schema evolution.