Databricks Machine Learning Associate Question 301
Select 4You are tasked with building a machine learning pipeline in Databricks using Spark ML to predict customer churn. The pipeline should include stages for data preparation, feature transformation, and model training. Which of the following components can be included as valid stages in a Spark ML Pipeline?
- A
VectorAssembler for combining feature columns into a single feature vector
- B
StringIndexer for converting categorical labels into numeric indices
- C
LinearRegression for training a regression model
- D
Pandas DataFrame for performing data transformations
- E
OneHotEncoder for encoding categorical features
- F
SQL queries for data preparation within the pipeline stages
Show answer and explanation
Correct answers: A, B, C, E
Explanation
Spark ML Pipelines consist of a sequence of stages, where each stage is either a Transformer (e.g., VectorAssembler, StringIndexer, OneHotEncoder) or an Estimator (e.g., LinearRegression). These stages operate on Spark DataFrames and are designed to work seamlessly within the pipeline framework. Pandas DataFrames and SQL queries, while useful for data manipulation, are not valid pipeline stages.
- A. Correct.
VectorAssembler is a valid Spark ML Pipeline stage used for combining multiple feature columns into a single vector, which is required by Spark ML algorithms.
- B. Correct.
StringIndexer is a valid Spark ML Pipeline stage used to convert categorical labels into numeric indices, a common preprocessing step in machine learning pipelines.
- C. Correct.
LinearRegression is a valid Spark ML Pipeline stage as it is a machine learning algorithm included in Spark ML for regression tasks.
- D. Incorrect.
Pandas DataFrame is not a valid Spark ML Pipeline stage because Spark ML pipelines operate on Spark DataFrames, not Pandas DataFrames.
- E. Correct.
OneHotEncoder is a valid Spark ML Pipeline stage for encoding categorical features as one-hot vectors, which is often used in feature transformation.
- F. Incorrect.
SQL queries are not directly supported as stages within Spark ML Pipelines, although SQL queries can be used outside the pipeline to prepare data.