Databricks Machine Learning Associate Question 297
Select 3You are tasked with building a machine learning pipeline in Spark ML to preprocess data and train a model. Which of the following statements correctly describe the role of an Estimator and a Transformer in this process?
- A
An Estimator represents a machine learning algorithm or a stage that can fit data to produce a Transformer.
- B
A Transformer takes a DataFrame as input and produces another DataFrame as output by applying some transformation.
- C
An Estimator directly modifies the input DataFrame to output predictions or transformed features.
- D
A Transformer always requires a training dataset to build its transformation logic.
- E
An Estimator is used during the training phase, while a Transformer is used during the transformation or prediction phase.
Show answer and explanation
Correct answers: A, B, E
Explanation
In Spark ML, an Estimator is a machine learning abstraction used to fit data and produce a Transformer. A Transformer, on the other hand, applies transformations or predictions to a DataFrame based on the learned logic. Estimators are typically used during training, while Transformers are used for transformations or predictions.
- A. Correct.
Correct. An Estimator in Spark ML represents a machine learning algorithm or pipeline stage that can fit on a DataFrame to produce a Transformer.
- B. Correct.
Correct. A Transformer in Spark ML takes a DataFrame as input and applies the transformation defined by its logic to produce another DataFrame.
- C. Incorrect.
Incorrect. An Estimator does not directly modify the input DataFrame, it first fits the data to produce a Transformer, which can then be used to modify the DataFrame.
- D. Incorrect.
Incorrect. A Transformer does not require a training dataset, as it uses the transformation logic already defined (often learned during the Estimator's fit phase).
- E. Correct.
Correct. An Estimator is used during the training phase to learn from data, while a Transformer is applied during the transformation or prediction phase.