Databricks Machine Learning Professional Question 228
Select 2A company has deployed a machine learning model to predict customer churn based on historical customer behavior. Recently, the company introduced a new pricing plan and updated its marketing strategy. After analyzing the model's performance, the team noticed a significant drop in accuracy. Which of the following scenarios are likely explanations for this issue?
- A
The distribution of customer demographic features has shifted due to the new marketing strategy.
- B
The relationship between customer behavior and churn has changed due to the introduction of the new pricing plan.
- C
The model's training data included customers who were not part of the new pricing plan.
- D
The model is overfitting to the training dataset and cannot generalize to new data.
- E
The model hyperparameters were not optimized during training.
Show answer and explanation
Correct answers: A, B
Explanation
Feature drift and label drift are common issues when significant changes occur in the data generating process. In this scenario, the new marketing strategy may have caused the distribution of input features to shift (feature drift), while the new pricing plan might have altered the relationship between customer behavior and the churn outcome (label drift). Recognizing these types of drift is crucial for maintaining model performance in dynamic environments.
- A. Correct.
Feature drift occurs when the distribution of input features changes over time. In this case, the new marketing strategy may have attracted a different segment of customers, leading to different demographic features being observed.
- B. Correct.
Label drift occurs when the relationship between the features and the target variable (churn) changes. The new pricing plan might have changed how customer behavior correlates with churn likelihood.
- C. Incorrect.
Although the training data may not include customers from the new pricing plan, this alone does not directly explain feature or label drift. Instead, it may indicate a lack of generalization to unseen scenarios rather than drift.
- D. Incorrect.
Overfitting is a model training issue and is unrelated to feature drift or label drift. It would not explain a sudden performance drop due to real-world changes in data.
- E. Incorrect.
Hyperparameter optimization is part of model training but is not directly related to feature drift or label drift. It would not explain performance degradation caused by real-world data changes.