Databricks Machine Learning Associate Question 299
Select 4You are tasked with building a machine learning pipeline in Databricks using Spark ML to predict customer churn. The dataset contains categorical and numerical features, and your pipeline must include the following steps: data preprocessing (handling missing values, encoding categorical variables, and scaling numerical features), a machine learning model, and evaluation of the model's performance. Which of the following steps are required to build such a pipeline?
- A
Use
VectorAssemblerto combine feature columns into a single feature vector. - B
Use
StringIndexerto encode categorical columns as numerical indices. - C
Use
StandardScalerto standardize numerical columns. - D
Directly fit a decision tree model on the raw dataset without any preprocessing.
- E
Add an evaluator like
RegressionEvaluatororBinaryClassificationEvaluatorto assess model performance.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
To build a machine learning pipeline in Spark ML, you need to preprocess the data by encoding categorical variables, scaling numerical features, and assembling them into a feature vector. Then, you can fit a machine learning model and evaluate its performance using an evaluator. Skipping any of these steps, such as preprocessing or evaluation, would result in an incomplete or non-functional pipeline.
- A. Correct.
Correct:
VectorAssembleris used to combine multiple feature columns into a single vector that Spark ML models can process. This is a required step for Spark ML pipelines. - B. Correct.
Correct:
StringIndexeris essential for converting categorical data into numerical representations, which are required by Spark ML models. - C. Correct.
Correct:
StandardScaleris commonly used to normalize numerical columns to ensure they have a mean of 0 and a standard deviation of 1, which improves model performance. - D. Incorrect.
Incorrect: Directly fitting a model on raw data without preprocessing will not work in Spark ML pipelines as models require features to be in a vectorized format, and categorical data must be encoded first.
- E. Correct.
Correct: Adding an evaluator like
RegressionEvaluatororBinaryClassificationEvaluatorallows you to assess the model's performance, which is a critical step in the pipeline.