Databricks Machine Learning Associate Question 300
Select 3You are tasked with building a machine learning pipeline in Databricks using Spark ML to predict customer churn. The pipeline consists of the following steps: data cleaning, feature engineering, training a classification model, and evaluation. Which of the following steps are correct when creating a pipeline using Spark ML?
- A
Define a sequence of stages where each stage is a Transformer or Estimator.
- B
Use a Pipeline object to chain the stages together.
- C
Directly call the
fitmethod on the data without defining any stages. - D
Use the
PipelineModelobject to apply the trained pipeline on new data. - E
Run the pipeline stages in parallel to optimize performance.
Show answer and explanation
Correct answers: A, B, D
Explanation
To create a pipeline in Spark ML, you define the stages as a sequence of Transformers and Estimators, chain them together using a Pipeline object, and fit the pipeline to your training data. The resulting PipelineModel can then be used to transform and predict on new data. Spark ML pipelines are designed to execute stages in a sequential manner to ensure proper data flow through the pipeline.
- A. Correct.
Correct. In Spark ML, a pipeline is defined as a sequence of stages, where each stage is either a Transformer or an Estimator.
- B. Correct.
Correct. A Pipeline object is used to chain the defined stages together into a coherent workflow.
- C. Incorrect.
Incorrect. You cannot call the
fitmethod on data without first defining and setting up the pipeline stages. - D. Correct.
Correct. Once the pipeline is trained, the resulting
PipelineModelis used to apply the transformations and predictions on new data. - E. Incorrect.
Incorrect. Pipeline stages are executed sequentially, not in parallel, as they may depend on intermediate outputs.