Databricks Machine Learning Professional Question 183
Single answerYou are developing a streaming application in Databricks using Structured Streaming to process real-time sensor data. The data may arrive out-of-order due to network latency or system delays. Which configuration or technique should you use to ensure that late-arriving data is handled appropriately?
- A
Set the 'watermark' parameter to define the allowed delay for late data.
- B
Use a 'groupByKey' operation to reorder the events based on their timestamps.
- C
Disable checkpointing to allow the system to process late data more flexibly.
- D
Increase the trigger interval to process batches of data less frequently.
Show answer and explanation
Correct answer: A
Explanation
Structured Streaming in Databricks allows you to handle late-arriving (out-of-order) data by using the 'watermark' parameter. Watermarking defines the maximum delay for which late data can be considered in aggregations or other operations. This ensures that the system can process late-arriving events within a defined timeframe while effectively managing state and performance.
- A. Correct.
The 'watermark' parameter allows you to specify how much delay is acceptable for late data. This is the correct approach for handling out-of-order data in Structured Streaming.
- B. Incorrect.
Using 'groupByKey' does not address the issue of late-arriving data. It is used to group data based on a key but does not help with defining acceptable lateness.
- C. Incorrect.
Disabling checkpointing does not help with handling late-arriving data. Checkpointing is used to store progress information for fault tolerance, and disabling it could lead to data loss.
- D. Incorrect.
Increasing the trigger interval affects the frequency of batch processing but does not solve the problem of handling out-of-order data.