Databricks Machine Learning Associate Question 303
Select 4You are developing a Spark ML Pipeline to train a model that predicts house prices. During development, you encounter unexpected behavior where the model performs poorly despite having a well-defined feature set. Which of the following factors should you check to address this issue?
- A
Ensure consistent column naming between the DataFrame and the pipeline stages
- B
Verify that all stages in the pipeline are fitted on the same training dataset
- C
Inspect and handle missing values in the dataset before fitting the pipeline
- D
Check for data leakage caused by using test data during pipeline fitting
- E
Confirm that the pipeline stages are executed in parallel for faster processing
Show answer and explanation
Correct answers: A, B, C, D
Explanation
When developing a Spark ML Pipeline, several common issues can lead to poor model performance. Ensuring consistent column naming across pipeline stages, fitting all stages on the same training dataset, handling missing values appropriately, and preventing data leakage are critical steps. These ensure that the pipeline operates correctly and produces reliable results. Parallel execution of pipeline stages is not applicable in Spark ML, so it is not relevant to the problem described.
- A. Correct.
Spark ML Pipelines rely on consistent column naming conventions for input and output columns across stages. If column names are mismatched, it can lead to runtime errors or incorrect results.
- B. Correct.
All pipeline stages must be fitted on the same training dataset. Mismatched datasets between stages can result in misaligned transformations and poor model performance.
- C. Correct.
Missing values can disrupt feature engineering and model training. These need to be handled before fitting the pipeline to avoid issues during transformation or training.
- D. Correct.
Data leakage occurs when test data is inadvertently used during pipeline fitting, leading to overly optimistic performance metrics and poor generalization to unseen data.
- E. Incorrect.
Pipeline stages in Spark ML are executed sequentially, not in parallel. Optimizing execution speed is unrelated to addressing model performance issues.