Databricks Machine Learning Associate Question 298
Select 4You are tasked with building a machine learning pipeline using Spark ML in Databricks to predict customer churn. The dataset includes both categorical and numerical features. Which of the following steps are necessary to correctly build and configure the pipeline?
- A
Use StringIndexer to encode categorical features into numerical format before including them in the pipeline.
- B
Normalize numerical features using MinMaxScaler to ensure they are on the same scale.
- C
Use VectorAssembler to combine all features into a single vector column named 'features'.
- D
Directly pass raw categorical columns to the machine learning model without encoding.
- E
Configure a Pipeline object with stages for feature transformations and the estimator.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
To build a robust Spark ML pipeline in Databricks, you must preprocess categorical features using StringIndexer, normalize numerical features to ensure consistent scaling, and assemble all features into a single vector column using VectorAssembler. These transformations, along with the machine learning model, are added as stages in a Pipeline object. Raw categorical data cannot be directly used in Spark ML models, so it must be encoded beforehand.
- A. Correct.
Correct: StringIndexer is required to convert categorical features into numerical format, as Spark ML models cannot handle raw categorical data.
- B. Correct.
Correct: Normalizing numerical features using MinMaxScaler ensures that they are on the same scale, improving the performance of many machine learning models.
- C. Correct.
Correct: VectorAssembler is necessary to combine all transformed features into a single vector column that Spark ML models can process.
- D. Incorrect.
Incorrect: Spark ML models cannot work with raw categorical columns. These must be encoded into numerical format first.
- E. Correct.
Correct: A Pipeline object is essential in Spark ML to define the sequence of feature transformations and the machine learning model (estimator).