AIF-C01 Question 57
Select 3A retail company has trained a machine learning model to predict customer churn. The model achieves 95% accuracy on the test dataset. However, when deployed, the business team notices that the cost of retaining churned customers is unexpectedly high. Which combination of metrics should the team focus on to better evaluate the model's effectiveness for their business goals?
- A
F1 score to balance precision and recall for churn predictions
- B
Area Under the ROC Curve (AUC) to evaluate the model's ability to rank customers by churn risk
- C
Cost per retained customer to assess the financial impact of retention strategies
- D
Model accuracy to ensure the model performs well on the dataset
- E
Return on Investment (ROI) to measure the profitability of the churn prediction system
Show answer and explanation
Correct answers: A, C, E
Explanation
Evaluating a model for business goals requires combining ML performance metrics like F1 score, which balances precision and recall, with business metrics like cost per retained customer and ROI. These metrics align with the company's concerns about high retention costs and ensure the model's predictions are both accurate and financially viable. Accuracy alone is insufficient for imbalanced datasets, and AUC does not directly address the financial impact.
- A. Correct.
F1 score is a critical metric when dealing with imbalanced datasets, such as churn prediction, ensuring the model balances false positives and false negatives effectively.
- B. Incorrect.
AUC is useful for ranking predictions, but it does not directly address the financial implications or business impact of the model's performance, which is the focus here.
- C. Correct.
Cost per retained customer is a business metric directly tied to the financial impact of the churn prediction system and aligns with the company's concerns about retention costs.
- D. Incorrect.
Accuracy is not sufficient for imbalanced datasets because a high accuracy might be misleading if the model simply predicts the majority class without effectively identifying churned customers.
- E. Correct.
ROI is essential for understanding the overall profitability and effectiveness of the churn prediction system from a business perspective.