Databricks Machine Learning Professional Question 59
Select 3You are building a machine learning pipeline in Databricks to predict customer churn. Your dataset contains several numerical features with varying scales, many missing values, and some categorical features with a high cardinality. Which of the following preprocessing steps should you include to prepare this dataset for training a gradient boosting model?
- A
Normalize the numerical features to have a mean of 0 and standard deviation of 1.
- B
Impute the missing values using the mean for numerical features and the mode for categorical features.
- C
One-hot encode all categorical features regardless of their cardinality.
- D
Use frequency encoding or target encoding for categorical features with high cardinality.
- E
Remove features with missing values exceeding a certain threshold (e.g., 70%).
Show answer and explanation
Correct answers: B, D, E
Explanation
Preparing a dataset for a gradient boosting model involves steps to handle missing values, address high-cardinality categorical features, and ensure the quality of the dataset. While normalization is unnecessary due to the scale-invariant nature of gradient boosting models, imputing missing values, encoding categorical features efficiently, and removing features with excessive missing values are critical to building an effective pipeline.
- A. Incorrect.
Normalization is not necessary for gradient boosting models because these models are scale-invariant, meaning they are not affected by differences in feature magnitudes.
- B. Correct.
Imputing missing values is an essential preprocessing step to ensure the dataset is complete before training. Using the mean for numerical features and the mode for categorical features is a standard approach.
- C. Incorrect.
One-hot encoding is not suitable for categorical features with high cardinality, as it can lead to a significant increase in the number of dimensions, making the model less efficient.
- D. Correct.
For categorical features with high cardinality, frequency encoding or target encoding is more efficient than one-hot encoding and helps avoid the curse of dimensionality while preserving important information.
- E. Correct.
Removing features with excessive missing values is a common preprocessing step to ensure that the dataset quality is maintained, especially when a feature has too many missing values to impute reliably.