Databricks Machine Learning Associate Question 307
Select 3While developing a Spark ML Pipeline in Databricks, a data scientist notices that the model's performance is poor in production despite good results during training. Which of the following issues might be causing this problem?
- A
The stages in the pipeline are not executed in the correct sequence.
- B
Data leakage occurred due to incorrect splitting of training and test data.
- C
Transformer stages in the pipeline were not persisted before deployment.
- D
The pipeline includes hardcoded column names that are unavailable during production inference.
- E
The model was evaluated on the same data used to train it.
Show answer and explanation
Correct answers: B, D, E
Explanation
When developing a Spark ML Pipeline, it is critical to avoid data leakage, ensure proper handling of column names for production inference, and maintain a clear separation between training and test data. These issues can lead to misleading training results and poor production performance. Understanding these 'gotchas' is essential for building robust pipelines.
- A. Incorrect.
Stages in a Spark ML Pipeline are executed in sequence as defined by the pipeline. If they were not executed correctly, the pipeline would fail entirely rather than degrade performance. This is not likely the cause of poor performance in production.
- B. Correct.
Data leakage can lead to overly optimistic training results but poor production performance because the model has indirectly 'seen' the test data during training.
- C. Incorrect.
While persisting transformer stages can help with reusability and efficiency, it does not directly affect model performance unless there are deployment-specific requirements. This is not a typical cause of performance degradation.
- D. Correct.
Hardcoded column names might not exist in the production data, leading to errors or incorrect feature engineering during inference, which can severely degrade model performance.
- E. Correct.
Evaluating the model on the same data as used for training can result in high training accuracy but does not provide a realistic performance measure. This leads to poor generalization and unexpected performance in production.