Databricks Machine Learning Associate Question 304
Select 3You are developing a Spark ML Pipeline in Databricks to train a machine learning model. During the pipeline's execution, you notice that the output contains unexpected results. Upon investigation, you find that the issue stems from the handling of categorical features. Which of the following practices could help prevent similar issues in Spark ML Pipelines?
- A
Ensure that StringIndexer is applied to all categorical columns before using them in machine learning models.
- B
Avoid reusing feature column names across different stages of the pipeline.
- C
Manually cast all numerical columns to DoubleType before feeding them into the VectorAssembler.
- D
Always use VectorAssembler to combine feature columns before passing them to the model.
- E
Verify that the pipeline stages are executed in the correct order during the transformation process.
Show answer and explanation
Correct answers: A, B, E
Explanation
Developing a Spark ML Pipeline requires careful attention to the handling of categorical features and the overall pipeline structure. Applying StringIndexer to categorical columns ensures they are numerically encoded, while avoiding feature name reuse prevents overwriting issues. Additionally, verifying the order of pipeline stage execution ensures proper transformations. These practices collectively help prevent common issues and ensure the pipeline produces expected results.
- A. Correct.
Correct: StringIndexer is essential for converting categorical columns into numerical indices required by Spark ML models.
- B. Correct.
Correct: Reusing feature column names across different stages can cause overwriting issues, leading to unexpected results in the pipeline.
- C. Incorrect.
Incorrect: While numerical columns should be of the correct type, Spark ML automatically handles numerical column conversions in many cases. Manual casting is not always necessary.
- D. Incorrect.
Incorrect: Although VectorAssembler is often used to combine feature columns, it is not mandatory for all Spark ML Pipelines, depending on the specific requirements.
- E. Correct.
Correct: Ensuring that pipeline stages are executed in the correct order is critical to avoid errors and achieve the desired transformations.