Databricks Machine Learning Associate Question 287
Select 3You are working on a distributed dataset in Spark ML to train a machine learning model. You decide to split the data into training and test sets using the randomSplit method. Which of the following considerations are important to ensure the split is performed correctly, especially in a distributed environment?
- A
The split ratios are approximate, not guaranteed, due to the nature of distributed data.
- B
The random seed should be set to ensure reproducibility across multiple runs.
- C
Data skew can occur when splitting, so ensuring an even distribution of data across partitions is essential.
- D
Using a very small dataset with
randomSplitensures perfect splits in distributed environments. - E
The order of rows in the dataset affects how
randomSplitpartitions the data.
Show answer and explanation
Correct answers: A, B, C
Explanation
When working with distributed data in Spark ML, it is important to account for the approximate nature of split ratios, ensure reproducibility by setting a random seed, and address potential data skew issues. These considerations help ensure that the training and test sets are appropriately split for model training and evaluation in a distributed environment.
- A. Correct.
Correct: In distributed systems, the
randomSplitmethod relies on approximate ratios due to data distribution, so the exact split ratios may not be guaranteed. - B. Correct.
Correct: Setting a random seed ensures consistent and reproducible splits across multiple runs, which is crucial for debugging and experimentation.
- C. Correct.
Correct: Data skew can affect the quality of splits in distributed systems, as some partitions may contain disproportionately large or small portions of the data.
- D. Incorrect.
Incorrect: A very small dataset is not ideal for
randomSplitin distributed environments, as distributed systems are optimized for larger datasets, and splits may not be reliable with inadequate data samples. - E. Incorrect.
Incorrect: The
randomSplitmethod is not dependent on the order of rows in the dataset; it uses random sampling to assign rows to splits.