Databricks Machine Learning Associate Question 306
Select 3While developing a Spark ML pipeline for a classification task, a data scientist encounters inconsistent results during model evaluation. After investigation, they identify that the issue arises from how the pipeline stages are defined. Which of the following are key considerations to ensure consistency in Spark ML pipelines?
- A
Ensure that the same random seed is set for all stages involving randomness, such as train-test splits or algorithms using stochastic processes.
- B
Always cache the DataFrame after each pipeline stage to ensure consistent processing.
- C
Verify that all stages in the pipeline are deterministic and do not depend on external data sources that may change.
- D
Ensure that the input schema of the DataFrame matches the expected schema for the pipeline stages, especially for stages like VectorAssembler.
- E
Use checkpointing to save intermediate pipeline results for every stage, regardless of the pipeline complexity.
Show answer and explanation
Correct answers: A, C, D
Explanation
To ensure consistency in Spark ML pipelines, it is important to account for randomness by setting a seed, ensure determinism by avoiding external dependencies, and validate schema compatibility for pipeline stages. These practices help avoid common issues like non-reproducible results, unexpected errors, or inconsistencies during pipeline execution. Caching and checkpointing, while useful in specific scenarios, are not general requirements for ensuring pipeline consistency.
- A. Correct.
Setting the same random seed ensures reproducibility for any operations or algorithms involving randomness, like splitting data or using stochastic solvers.
- B. Incorrect.
Caching DataFrames after every stage is unnecessary and can lead to inefficient memory usage. It is not a required step for consistency in Spark ML pipelines.
- C. Correct.
Pipeline stages should be deterministic and independent of external data sources that can change between runs, as this ensures consistent behavior across executions.
- D. Correct.
Ensuring the input schema matches the pipeline stage requirements, particularly for stages like VectorAssembler, is critical to avoid runtime errors or unexpected behavior.
- E. Incorrect.
While checkpointing can be useful in certain scenarios, it is not required for ensuring consistency in Spark ML pipelines unless the pipeline is too large to fit in memory or has very complex stages.