Databricks Machine Learning Professional Question 255
Select 2A team is monitoring a machine learning model used for predicting customer churn. One of the model's features is a categorical variable representing customer subscription types (e.g., 'Basic', 'Premium', 'Enterprise'). The team notices potential drift in the distribution of this feature between training and production data. Why might statistical tests, such as a Chi-Square test, provide a more robust monitoring solution for categorical feature drift than relying solely on summary statistics like mode or category proportions?
- A
Statistical tests can detect significant changes in category distributions that might not be obvious from summary statistics.
- B
Statistical tests can identify correlations between multiple categorical features, while summary statistics cannot.
- C
Summary statistics like mode only focus on the most frequent category and may miss rare but important shifts in other categories.
- D
Statistical tests are designed to monitor feature drift continuously, while summary statistics can only be calculated periodically.
Show answer and explanation
Correct answers: A, C
Explanation
Statistical tests, such as the Chi-Square test, are more robust for monitoring drift in categorical features because they evaluate the entire distribution rather than focusing on individual summary measures like mode or category proportions. This ensures that even rare but significant changes in category distributions are captured, which might be missed by simpler summary statistics.
- A. Correct.
Statistical tests like the Chi-Square test can evaluate whether the entire distribution of a categorical feature has shifted significantly, capturing changes across all categories, not just the most frequent ones. This makes them more robust for monitoring drift.
- B. Incorrect.
This is incorrect because statistical tests such as Chi-Square are not used to analyze correlations between features, they focus on individual distributions.
- C. Correct.
Summary statistics like mode only describe the most frequent category and ignore shifts in less frequent categories. Statistical tests evaluate all categories collectively, making them more thorough.
- D. Incorrect.
This is incorrect because both statistical tests and summary statistics can be computed periodically or continuously. The key difference lies in the depth of insights they provide, not the timing of their application.