Databricks Machine Learning Associate Question 285
Select 2You are working on a distributed dataset in Databricks and need to split it into training, validation, and test sets using Spark ML's randomSplit method. Which of the following considerations must you keep in mind to ensure data is split correctly and reproducibly?
- A
The seed parameter should be set to ensure reproducibility of splits.
- B
The sum of the weights provided in
randomSplitmust equal 1.0. - C
Data skew in the dataset can lead to uneven splits across partitions.
- D
randomSplitmay result in overlapping data points across splits without proper configuration. - E
The dataset should always be cached before performing a
randomSplitto avoid recomputation.
Show answer and explanation
Correct answers: A, C
Explanation
When using randomSplit in Spark ML, setting a seed ensures reproducibility of the splits, which is essential for consistent experimentation. Additionally, in distributed systems, data skew can impact how data is partitioned and lead to imbalanced splits. Understanding these considerations helps in building reliable machine learning pipelines.
- A. Correct.
Setting the seed parameter ensures that the splits are reproducible, which is critical for consistent results when rerunning workflows.
- B. Incorrect.
This is incorrect because the weights provided in
randomSplitdo not need to sum to 1.0; they are just relative proportions. - C. Correct.
Data skew can lead to unevenly distributed data across partitions, which may result in imbalanced splits.
- D. Incorrect.
randomSplitensures that no data point is included in multiple splits, so this is not a concern. - E. Incorrect.
While caching can improve performance, it is not a strict requirement for
randomSplitto function correctly.