Databricks Machine Learning Professional Question 182
Select 2You are building a real-time fraud detection system using Databricks Structured Streaming. The incoming transaction data can sometimes arrive out-of-order due to network delays. Which of the following features of Structured Streaming can help you handle out-of-order data effectively?
- A
Watermarking to manage late data
- B
Stateful operations to track and aggregate information over time
- C
Checkpointing to recover the streaming query in case of failure
- D
Event-time processing to order data based on timestamps
- E
Trigger intervals to control the data output frequency
Show answer and explanation
Correct answers: A, D
Explanation
In Structured Streaming, out-of-order data can be effectively managed using watermarking, which defines a threshold for late-arriving data, and event-time processing, which ensures data is ordered based on timestamps rather than arrival time. These features are specifically designed to handle scenarios where data does not arrive in sequential order.
- A. Correct.
Watermarking allows the system to define how much late data it can tolerate and still process, making it essential for handling out-of-order data.
- B. Incorrect.
Stateful operations are useful for maintaining and processing stateful data but do not directly address out-of-order data.
- C. Incorrect.
Checkpointing is crucial for fault tolerance and recovery but does not handle out-of-order data.
- D. Correct.
Event-time processing ensures that data is processed in the order of their event timestamps, making it integral to managing out-of-order data.
- E. Incorrect.
Trigger intervals determine the frequency of output and do not directly address the issue of out-of-order data.