Databricks Data Engineer Professional Question 78
Select 3You are designing a real-time data ingestion pipeline using Structured Streaming and Delta Lake. The pipeline processes clickstream data from a website and needs to support late-arriving events while ensuring exactly-once processing guarantees. Which combination of design patterns should you implement to meet the requirements?
- A
Use Delta Lake's MERGE operation to handle late-arriving events and update records accordingly.
- B
Use watermarking in Structured Streaming to gracefully manage late-arriving events.
- C
Enable delta.autoOptimize.optimizeWrite to ensure exactly-once processing.
- D
Use checkpointing in Structured Streaming to maintain state and achieve fault tolerance.
- E
Avoid using a primary key in Delta Lake to improve performance for late-arriving events.
Show answer and explanation
Correct answers: A, B, D
Explanation
To meet the requirements of handling late-arriving events and ensuring exactly-once processing, a combination of Delta Lake's MERGE operation, Structured Streaming's watermarking, and checkpointing is required. MERGE allows for upserts, watermarking manages late data, and checkpointing ensures state and fault tolerance, making this trio of design patterns ideal for the scenario.
- A. Correct.
Delta Lake's MERGE operation is specifically designed for updating or inserting records (upserts), which is essential for handling late-arriving events in streaming scenarios.
- B. Correct.
Watermarking allows Structured Streaming to define a threshold for late-arriving events, enabling efficient and controlled handling of such events.
- C. Incorrect.
While delta.autoOptimize.optimizeWrite optimizes write operations for performance, it does not directly contribute to exactly-once processing guarantees.
- D. Correct.
Checkpointing in Structured Streaming is a critical mechanism for maintaining state and fault tolerance, which is essential for ensuring exactly-once processing guarantees.
- E. Incorrect.
Avoiding a primary key in Delta Lake may degrade the ability to efficiently handle updates required for late-arriving events, as a primary key is often needed for MERGE operations.