Databricks Machine Learning Professional Question 243
Select 3A machine learning team has deployed a model that uses a numeric feature called average_monthly_expense. Over time, they suspect that the distribution of this feature in incoming data has changed, which could degrade model performance. They want to use summary statistic monitoring as a simple solution to detect feature drift. Which of the following monitoring approaches aligns with this method?
- A
Track changes in the mean and standard deviation of
average_monthly_expenseover time. - B
Monitor the correlation of
average_monthly_expensewith the target variable. - C
Compare the median and interquartile range (IQR) of
average_monthly_expensein recent data to those from the training data. - D
Use a Kolmogorov-Smirnov (KS) test to statistically compare the full distributions of
average_monthly_expensein training and production data. - E
Set thresholds for acceptable ranges of summary statistics such as mean, median, and standard deviation for
average_monthly_expense.
Show answer and explanation
Correct answers: A, C, E
Explanation
Summary statistic monitoring for numeric feature drift involves tracking simple, aggregated metrics like mean, median, standard deviation, or IQR over time. These metrics provide a lightweight and interpretable way to detect shifts in feature distribution compared to the training data. This approach is especially useful for production monitoring because it is computationally efficient and easy to implement with predefined thresholds.
- A. Correct.
Correct: Monitoring changes in summary statistics like the mean and standard deviation is a simple and effective way to detect numeric feature drift over time.
- B. Incorrect.
Incorrect: Monitoring correlation with the target variable is useful for feature importance analysis but is not directly related to detecting drift via summary statistics.
- C. Correct.
Correct: Comparing robust statistics like the median and IQR between training and production data provides insight into potential drift, especially when the feature distribution is skewed or has outliers.
- D. Incorrect.
Incorrect: While the Kolmogorov-Smirnov (KS) test is a valid statistical method for detecting distributional differences, it is not considered a 'summary statistic' approach and is more computationally intensive.
- E. Correct.
Correct: Setting thresholds for summary statistics (e.g., acceptable mean, median, or standard deviation ranges) is a practical way to monitor feature drift in production.