Databricks Machine Learning Professional Question 267
Select 4You are a data scientist managing a machine learning model deployed in production for predicting customer churn. You notice a significant drop in model performance over the last month. After investigating, you suspect both data drift and concept drift may have occurred. Which of the following steps should you take to build a comprehensive drift detection and mitigation strategy in Databricks?
- A
Implement a monitoring system to track feature distributions over time and compare them to training data distributions.
- B
Train a shadow model on the latest production data and compare its performance to the original model.
- C
Regularly re-train your model on the same training dataset to ensure consistency.
- D
Analyze the correlation between features and the target variable in the new data to detect concept drift.
- E
Set up a batch inference pipeline to log model predictions and monitor prediction distributions over time.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Comprehensive drift solutions in Databricks involve monitoring both data drift (changes in input data distributions) and concept drift (changes in the relationship between input features and the target variable). Steps such as tracking feature distributions, analyzing feature-target correlations, and monitoring predictions over time help detect drift, while training a shadow model can validate the presence of concept drift. Simply re-training on the original dataset does not address drift issues, as it does not consider changes in the data or relationships over time.
- A. Correct.
Monitoring feature distributions is a key step in detecting data drift, as it allows you to identify changes in input data characteristics over time.
- B. Correct.
Using a shadow model trained on recent production data helps detect concept drift by comparing how well the new data aligns with the original model's logic.
- C. Incorrect.
Re-training the model on the same training dataset does not address drift, as it does not incorporate the changes in the data or the target concept.
- D. Correct.
Analyzing the correlation between features and the target variable can help detect concept drift, as it identifies changes in the relationship between features and the outcome.
- E. Correct.
Monitoring prediction distributions over time helps detect both data and concept drift, as shifts in predictions can indicate underlying issues in input data or model understanding.