Databricks Machine Learning Professional Question 269
Select 4You are tasked with building a pipeline in Databricks to monitor both concept drift and feature drift in a deployed machine learning model. Which steps should be included in your workflow to effectively measure these types of drift?
- A
Periodically compute statistical metrics, such as mean and variance, for important features in new data and compare them to the training data.
- B
Use a model monitoring tool to evaluate the performance metrics (e.g., accuracy, precision) of the deployed model on new data over time.
- C
Randomly shuffle the feature values in the new data to check for any inconsistencies with the training data distribution.
- D
Train a proxy model to distinguish between the training data and new data and analyze its performance as a measure of drift.
- E
Monitor the distribution of model prediction probabilities and compare them to the distribution observed during training.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Measuring concept drift and feature drift requires monitoring both changes in feature distributions (feature drift) and changes in the relationship between features and target variables (concept drift). Statistical metrics, performance monitoring, proxy models, and prediction probability distributions are essential tools to identify and quantify these changes. Randomly shuffling data, however, does not contribute to drift measurement and is not a valid approach.
- A. Correct.
This is correct. Computing statistical metrics like mean and variance for features can help identify feature drift by detecting changes in feature distributions over time.
- B. Correct.
This is correct. Monitoring performance metrics of the deployed model helps detect concept drift, as a drop in performance often indicates a change in the underlying data relationships.
- C. Incorrect.
This is incorrect. Randomly shuffling feature values does not provide meaningful insights into feature or concept drift. It introduces noise rather than identifying changes in distributions.
- D. Correct.
This is correct. Training a proxy model to distinguish between training and new data is a common technique for measuring drift, as a high-performing proxy model indicates significant differences between the datasets.
- E. Correct.
This is correct. Monitoring prediction probability distributions can reveal concept drift by showing if the model is producing predictions that differ from those observed during training.