Databricks Machine Learning Associate Question 496
Select 3You are working on a machine learning project in Databricks and need to prepare a dataset for training. The dataset contains missing values, categorical features, and outliers. Which of the following steps are appropriate to include in your data processing pipeline to address these issues?
- A
Impute missing values using a statistical method such as mean or median.
- B
Remove any rows with missing values to avoid introducing bias.
- C
Use one-hot encoding to transform categorical features into numerical format.
- D
Apply normalization or standardization to numerical features with outliers.
- E
Randomly shuffle the dataset without addressing missing values or outliers.
Show answer and explanation
Correct answers: A, C, D
Explanation
In a machine learning pipeline, handling missing values, transforming categorical features, and addressing outliers are crucial steps to ensure the dataset is clean and suitable for model training. Imputation (Option 1) prevents data loss due to missing values, one-hot encoding (Option 3) transforms categorical data into numerical format, and normalization or standardization (Option 4) mitigates the impact of outliers. Removing rows with missing values (Option 2) can lead to biased datasets, and random shuffling without data cleaning (Option 5) does not address the core issues.
- A. Correct.
Imputing missing values using a statistical method such as mean or median is a common and effective approach to handle missing data without discarding too much information.
- B. Incorrect.
Removing rows with missing values can result in a significant loss of data and may introduce bias, especially if the missing data is not random.
- C. Correct.
One-hot encoding is a standard technique to convert categorical features into a format suitable for machine learning models.
- D. Correct.
Normalization or standardization helps in handling numerical features with outliers by transforming their range or distribution, which can improve model performance.
- E. Incorrect.
Randomly shuffling the dataset without addressing missing values or outliers is not a valid data processing step as it does not resolve any of the identified issues.