Databricks Machine Learning Professional Question 250
Single answerYou are monitoring a machine learning model in production and notice that a numeric feature's distribution may be changing over time. Why might hypothesis tests, such as a Kolmogorov-Smirnov (KS) test, be a more robust solution for detecting feature drift compared to monitoring simple summary statistics like mean or standard deviation?
- A
Hypothesis tests can detect changes in the entire distribution, while summary statistics only capture specific aspects like central tendency or spread.
- B
Hypothesis tests are computationally faster than calculating summary statistics like mean or standard deviation.
- C
Hypothesis tests allow you to quantify the statistical significance of drift, which summary statistics cannot provide.
- D
Summary statistics are sufficient to detect all types of feature drift, so hypothesis tests are unnecessary.
Show answer and explanation
Correct answer: A
Explanation
Hypothesis tests are a more robust solution for detecting numeric feature drift because they analyze the entire feature distribution rather than focusing on specific aspects like the mean or standard deviation. This allows them to detect subtle or localized changes in the distribution that summary statistics may overlook. Moreover, while hypothesis tests can quantify statistical significance, their robustness primarily stems from evaluating distribution-wide changes rather than isolated metrics.
- A. Correct.
Correct: Hypothesis tests, such as the Kolmogorov-Smirnov test, analyze changes across the entire feature distribution, making them more robust for detecting nuanced changes that summary statistics like mean or standard deviation may miss.
- B. Incorrect.
Incorrect: Hypothesis tests are typically more computationally expensive than calculating simple summary statistics, as they involve comparing distributions rather than calculating single numbers.
- C. Incorrect.
Incorrect: While hypothesis tests do quantify statistical significance, this is not the primary reason they are more robust than summary statistics in detecting feature drift.
- D. Incorrect.
Incorrect: Summary statistics can fail to detect certain changes, such as shifts in the tail of a distribution or multi-modal changes, making them insufficient for comprehensive drift detection.