Databricks Machine Learning Associate Question 305
Select 3You are developing a Spark ML pipeline to predict customer churn using Databricks. During the development process, you encounter issues where the pipeline fails at runtime when applied to new data. Which of the following practices could help avoid such issues?
- A
Ensure that the transformations in the pipeline handle missing or unseen categories in the input data.
- B
Use a consistent schema for all datasets (training, validation, and testing) that will pass through the pipeline.
- C
Skip the step of persisting intermediate DataFrames, as Spark ML pipelines automatically optimize memory usage.
- D
Test the pipeline on a small batch of unseen data to identify potential edge cases and runtime errors.
- E
Avoid using feature scaling in the pipeline, as it can lead to runtime errors when encountering new data.
Show answer and explanation
Correct answers: A, B, D
Explanation
When developing a Spark ML pipeline, runtime errors can often occur due to schema mismatches, unhandled edge cases, or transformations failing to process unseen data. Ensuring the pipeline is robust to unseen categories, using consistent schemas, and testing on unseen data helps identify and mitigate these issues. Avoiding practices like skipping schema checks or neglecting testing can lead to runtime failures.
- A. Correct.
Correct. Spark ML pipelines may fail if transformations (e.g., one-hot encoding) can't handle unseen categories or missing values in new data. Implementing robust handling is critical.
- B. Correct.
Correct. Consistent schemas ensure that the pipeline can process data without schema mismatches causing runtime errors.
- C. Incorrect.
Incorrect. Persisting intermediate DataFrames can help optimize performance and prevent recomputation, but it is not directly related to avoiding runtime failures.
- D. Correct.
Correct. Testing the pipeline on a small batch of unseen data can help identify runtime issues, such as schema mismatches or unhandled edge cases.
- E. Incorrect.
Incorrect. Feature scaling is an essential part of many machine learning pipelines and does not inherently cause runtime errors. Issues arise only if scaling isn't applied consistently across all data splits.