Databricks Data Engineer Professional Question 73
Select 3You are designing a real-time data pipeline using Databricks Structured Streaming and Delta Lake. The pipeline must ensure exactly-once processing semantics for streaming data and maintain a history of changes for audit purposes. Which of the following design patterns should you implement to meet these requirements?
- A
Use Delta Lake's MERGE INTO operation to upsert data into a Delta table
- B
Enable checkpointing in Structured Streaming to track progress
- C
Use Delta Lake's Time Travel feature to query historical data
- D
Manually manage data consistency by using external storage for intermediate results
- E
Write streaming data directly into Delta Lake tables with append mode
Show answer and explanation
Correct answers: A, B, C
Explanation
To achieve exactly-once processing semantics and maintain a history of changes for audit purposes, combining Delta Lake's transactional capabilities (e.g., MERGE INTO for upserts) with Structured Streaming's checkpointing ensures fault tolerance and data consistency. Additionally, Delta Lake's Time Travel feature provides the ability to query historical data, meeting the auditability requirement.
- A. Correct.
Correct: Delta Lake's MERGE INTO operation is useful for upserting data to ensure exactly-once processing and handling updates/deletes in the stream.
- B. Correct.
Correct: Checkpointing in Structured Streaming ensures fault tolerance by storing progress information, which is crucial for exactly-once semantics.
- C. Correct.
Correct: Delta Lake's Time Travel feature allows querying historical states of data, which supports the auditability requirement.
- D. Incorrect.
Incorrect: Manually managing data consistency using external storage is error-prone and not necessary when using Delta Lake, which provides built-in transaction guarantees.
- E. Incorrect.
Incorrect: Writing streaming data with append mode does not support updates or deletions and does not ensure exactly-once semantics.