Databricks Machine Learning Professional Question 268
Select 3A data science team is operating a machine learning model in production that predicts customer churn. They suspect that the input data distribution has changed over time, leading to a potential decline in model performance. Which of the following steps should be included in a workflow to measure both concept drift and feature drift?
- A
Compare the statistical distributions of input features in the training data and recent production data.
- B
Use performance metrics (e.g., accuracy, precision, recall) to evaluate the model on recent production data with ground truth labels.
- C
Generate synthetic data to test the model's robustness to out-of-distribution inputs.
- D
Apply a drift detection algorithm (e.g., Kolmogorov-Smirnov test) to evaluate changes in feature distributions.
- E
Log model inference latency to determine if computational drift is affecting predictions.
Show answer and explanation
Correct answers: A, B, D
Explanation
To measure concept drift and feature drift, the workflow should include steps to compare feature distributions between training and production data, evaluate model performance on production data, and apply drift detection algorithms to quantify changes in feature distributions. These steps together help ensure that both changes in input data and the relationship between inputs and outputs are effectively monitored.
- A. Correct.
Comparing the statistical distributions of input features between training and production data is a key step in identifying feature drift, which occurs when input feature distributions change over time.
- B. Correct.
Evaluating the model's performance on recent production data with ground truth labels is essential for detecting concept drift, which refers to changes in the relationship between input features and the target variable.
- C. Incorrect.
While generating synthetic data can help test model robustness, it is not directly relevant to measuring concept or feature drift.
- D. Correct.
Drift detection algorithms, such as the Kolmogorov-Smirnov test, are useful for quantifying and identifying changes in feature distributions, which is critical for detecting feature drift.
- E. Incorrect.
Logging model inference latency is unrelated to concept or feature drift; it pertains to monitoring system performance rather than data or model behavior.