Databricks Machine Learning Professional Question 186
Single answerA retail company uses a Databricks streaming deployment to predict the daily demand for various products in real-time. The predictions are generated continuously and stored in a time-based prediction table. Which of the following approaches would most effectively identify anomalies in the continuous predictions over time?
- A
Use a window-based aggregation to compute the rolling mean and standard deviation of the predictions and flag significant deviations.
- B
Apply a clustering algorithm like K-Means on the predictions to identify clusters of anomalous predictions.
- C
Export the predictions to an external system for offline analysis and anomaly detection.
- D
Train a neural network model on historical demand data and use it to replace the predictions in the streaming table.
Show answer and explanation
Correct answer: A
Explanation
In a streaming deployment with continuous predictions stored in a time-based prediction table, real-time anomaly detection is critical. Window-based aggregation allows for efficient computation of statistical metrics such as rolling mean and standard deviation over a time window, enabling the system to identify significant deviations in predictions as potential anomalies. This approach is both time-efficient and well-integrated with the Databricks streaming architecture.
- A. Correct.
This is the correct option. Window-based aggregation is well-suited for time-based prediction data in streaming deployments. It enables real-time anomaly detection by monitoring deviations from expected patterns within a defined time window.
- B. Incorrect.
Clustering algorithms like K-Means are not ideal for identifying anomalies in continuous time-series predictions because they are better suited for grouping data points into clusters and do not account for temporal dependencies.
- C. Incorrect.
Exporting predictions for offline analysis does not align with the goal of real-time anomaly detection in streaming deployments. This approach introduces delays and negates the benefits of continuous predictions.
- D. Incorrect.
Replacing predictions with a neural network model does not address anomaly detection. Instead, it focuses on generating predictions, which is outside the scope of the given scenario.