Databricks Machine Learning Associate Question 296
Select 2You are building a machine learning pipeline in Databricks using Spark ML. As part of the pipeline, you use a VectorAssembler to combine multiple feature columns into a single feature vector and a LogisticRegression model for binary classification. Which of the following correctly describes the roles of VectorAssembler and LogisticRegression in the Spark ML pipeline?
- A
VectorAssembler is an Estimator because it takes raw data and generates a feature vector.
- B
VectorAssembler is a Transformer because it transforms input DataFrame columns into a single combined feature column.
- C
LogisticRegression is an Estimator because it learns a model based on training data and produces a Transformer.
- D
LogisticRegression is a Transformer because it transforms input features into predictions without learning a model.
Show answer and explanation
Correct answers: B, C
Explanation
In Spark ML, Transformers apply a deterministic transformation to a DataFrame without learning any parameters, while Estimators are responsible for learning parameters based on training data and producing Transformers. VectorAssembler is a Transformer because it combines input columns into a single feature vector without learning. LogisticRegression is an Estimator because it trains a model and outputs a Transformer capable of making predictions.
- A. Incorrect.
Incorrect. VectorAssembler is not an Estimator; it does not learn a model or produce another Transformer. Instead, it directly transforms input DataFrame columns into a new feature vector column.
- B. Correct.
Correct. VectorAssembler is a Transformer because it applies a deterministic transformation to input DataFrame columns and outputs a new column without learning from the data.
- C. Correct.
Correct. LogisticRegression is an Estimator because it involves a training process to learn a model (e.g., weights and biases) and produces a Transformer (e.g., a model that can make predictions).
- D. Incorrect.
Incorrect. LogisticRegression is not a Transformer. It undergoes training to learn a model, which makes it an Estimator. The resulting model is the Transformer that applies predictions.