Databricks Machine Learning Professional Question 180
Single answerYou are working on a machine learning pipeline using Structured Streaming in Databricks. Your streaming data source is a Kafka topic that occasionally produces events with timestamps that are not in order due to network delays. What approach can you use to handle this scenario effectively?
- A
Use watermarking to handle late-arriving data and define a threshold for allowed lateness.
- B
Increase the streaming trigger interval to ensure all data arrives in order before processing.
- C
Ignore late-arriving data since Structured Streaming automatically adjusts for out-of-order events.
- D
Switch to a batch processing model to avoid issues caused by out-of-order data.
Show answer and explanation
Correct answer: A
Explanation
In Structured Streaming, out-of-order data is a common challenge when dealing with real-time data sources like Kafka. Watermarking is a feature that allows you to define a threshold for late-arriving data, enabling the system to handle events that arrive out of order within a specified time window. This approach ensures that your streaming application processes events correctly while discarding excessively late data.
- A. Correct.
Correct. Watermarking allows you to specify a time threshold for late-arriving data, ensuring that events arriving out of order within this threshold are processed, while older events are discarded.
- B. Incorrect.
Incorrect. Increasing the streaming trigger interval will not guarantee that all data arrives in order, as out-of-order events can still occur due to other factors such as network delays.
- C. Incorrect.
Incorrect. Structured Streaming does not automatically adjust for out-of-order events. You need to explicitly handle late-arriving data using watermarking or similar techniques.
- D. Incorrect.
Incorrect. Switching to batch processing is not necessary for handling out-of-order events in Structured Streaming. Batch processing might introduce unnecessary latency for real-time applications.