Databricks Machine Learning Professional Question 271
Select 3You are working on a machine learning pipeline in Databricks to monitor a deployed model for potential performance degradation over time. The business team is concerned about both concept drift (changes in the relationship between features and the target) and feature drift (changes in the distribution of input features). What is a common workflow to measure and monitor these types of drift?
- A
Calculate the statistical distance (e.g., KL divergence) between the feature distributions in the training data and recent production data.
- B
Use a prequential approach to continuously evaluate the model on incoming production data and track performance metrics such as accuracy or F1-score.
- C
Periodically retrain the model on the production data without analyzing drift to ensure it adapts to new data.
- D
Implement a data monitoring system to log the distributions of features and target variables over time for comparison.
- E
Directly compare the model weights from the original training phase with the weights after retraining on new production data.
Show answer and explanation
Correct answers: A, B, D
Explanation
A common workflow for measuring concept drift and feature drift involves calculating statistical metrics (e.g., KL divergence) to detect feature drift, using prequential methods to monitor changes in model performance for concept drift, and implementing a data monitoring system to log distributions of features and targets over time. These approaches together provide a comprehensive way to identify and address drift in a machine learning pipeline.
- A. Correct.
Calculating statistical distance, such as KL divergence, is a common method to detect feature drift because it quantifies how much the feature distributions in production differ from the training data.
- B. Correct.
Using a prequential approach to continuously evaluate the model on production data helps identify concept drift by monitoring changes in performance metrics over time.
- C. Incorrect.
Retraining the model without analyzing drift does not measure or monitor drift but simply attempts to adapt to new data. This is not a recommended workflow for drift detection.
- D. Correct.
A data monitoring system that logs feature and target distributions over time is essential for identifying both feature and concept drift by enabling comparisons between training and production data.
- E. Incorrect.
Comparing model weights does not provide a direct measure of drift in the data or target relationships. It is an indirect and less effective method for detecting drift.