Databricks Machine Learning Associate Question 284
Select 4You are building a machine learning model using Spark ML on a large distributed dataset. While splitting the dataset into training and test sets using the randomSplit method, which of the following should you carefully consider to ensure the split is performed correctly?
- A
The seed value used for random number generation
- B
The total size of the dataset and its distribution across partitions
- C
Ensuring the sum of the split ratios equals 1.0
- D
The possibility of overlapping data between splits when performing random splits in Spark
- E
The use of stratified sampling to maintain the class distribution in both splits
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Splitting distributed data in Spark ML involves considerations like reproducibility (seed value), the underlying data distribution across partitions, and potential issues such as overlapping data between splits. Stratified sampling is crucial for maintaining class balance in imbalanced datasets. While Spark normalizes split ratios internally, this is not a critical issue to address explicitly.
- A. Correct.
Using a consistent seed value ensures reproducibility of the dataset split. Without it, the split may vary each time, leading to inconsistent results during testing and training.
- B. Correct.
If the dataset is imbalanced across partitions, some partitions may contain too few rows, which could cause bias or errors in the split.
- C. Incorrect.
While it is not mandatory for the sum of the split ratios to equal 1.0, Spark will normalize them internally if they don’t. This option is not a critical consideration.
- D. Correct.
In distributed systems like Spark, random splits may result in overlapping data between splits due to how the split logic is implemented. This can introduce data leakage and must be accounted for.
- E. Correct.
Stratified sampling ensures that the class distribution is preserved in both training and test sets, particularly important for imbalanced datasets. Failing to use stratified sampling may lead to biased models.