Databricks Machine Learning Associate Question 294
Select 4You are working on a machine learning pipeline in Databricks using Spark ML. You have a dataset with several numerical features and need to standardize these features before training a linear regression model. Which of the following correctly describes the roles of a Spark ML Estimator and a Spark ML Transformer in this scenario?
- A
The StandardScaler is an Estimator that learns the mean and standard deviation of the dataset to create a transformation model.
- B
The StandardScalerModel is a Transformer that applies the scaling transformation to the dataset.
- C
The LinearRegression is an Estimator that creates a model by fitting the training data.
- D
The LinearRegressionModel is a Transformer that transforms the input data into predictions.
- E
The StandardScalerModel is an Estimator that learns the transformation parameters for scaling the dataset.
Show answer and explanation
Correct answers: A, B, C, D
Explanation
In Spark ML, an Estimator is responsible for learning parameters from the data and producing a model (Transformer). For example, StandardScaler (an Estimator) computes the mean and standard deviation, and then produces StandardScalerModel (a Transformer). Similarly, LinearRegression (an Estimator) fits a linear model to the data and produces LinearRegressionModel (a Transformer) to make predictions. Understanding the distinction between Estimator and Transformer is critical for building and debugging Spark ML pipelines.
- A. Correct.
Correct: The StandardScaler is an Estimator because it fits on the dataset to compute the mean and standard deviation, which are the parameters needed for scaling.
- B. Correct.
Correct: The StandardScalerModel is a Transformer because it uses the learned parameters (mean and standard deviation) to scale the dataset.
- C. Correct.
Correct: The LinearRegression is an Estimator as it fits a linear model based on the training data.
- D. Correct.
Correct: The LinearRegressionModel is a Transformer because it takes input data and transforms it into predictions using the learned model.
- E. Incorrect.
Incorrect: The StandardScalerModel is not an Estimator. It does not learn parameters; it only applies the transformation using already learned parameters.