Databricks Data Engineer Professional Question 75
Select 3You are designing a data pipeline that ingests real-time streaming data into a Delta Lake table for downstream analytics. The incoming data includes late-arriving events, and the business requires accurate daily aggregates to be generated. Which of the following design patterns should you implement to meet these requirements?
- A
Use watermarking to handle late-arriving data and ensure only relevant data is processed within a time window.
- B
Write streaming data directly into a Delta table, and use Delta Lake's MERGE INTO functionality to update records based on event time.
- C
Schedule a daily batch job to overwrite the Delta table with the latest data, ensuring data consistency.
- D
Leverage Delta Lake's time travel feature to query historical data and recompute aggregates as needed.
- E
Use Structured Streaming's foreachBatch to apply custom logic for aggregations and write results into Delta tables.
Show answer and explanation
Correct answers: A, B, D
Explanation
To build an effective pipeline for real-time streaming data with late-arriving events, you should combine Structured Streaming's watermarking for time-window-based processing, Delta Lake's MERGE INTO for handling updates due to late events, and Delta Lake's time travel feature for recomputing aggregates or validating historical data. These design patterns ensure accuracy, consistency, and scalability in the pipeline.
- A. Correct.
Watermarking is a common design pattern in Structured Streaming to handle late-arriving data by defining the maximum delay allowed for an event to be considered in a specific time window. This ensures accuracy in processing while maintaining performance.
- B. Correct.
Delta Lake's MERGE INTO functionality is useful for handling updates or upserts, particularly for late-arriving events. This allows you to ensure that the data in the Delta table remains accurate and consistent.
- C. Incorrect.
While a daily batch job could overwrite data, this approach is not efficient or reliable for handling late-arriving events. It does not leverage the real-time capabilities of Structured Streaming and Delta Lake.
- D. Correct.
Delta Lake's time travel feature allows querying historical snapshots of data, which can be used to recompute aggregates or verify data accuracy when late-arriving events are processed.
- E. Incorrect.
The foreachBatch approach is useful when custom logic needs to be applied to each micro-batch, but it is not inherently designed for managing late-arriving events or daily aggregates in this scenario.