Databricks Machine Learning Associate Question 286
Select 3When splitting a distributed dataset in Spark ML for training and testing, which of the following considerations should you keep in mind to ensure a proper split?
- A
Ensure that the dataset is shuffled before splitting to avoid skewed splits.
- B
Use the same random seed across all splits to ensure reproducibility.
- C
Split the dataset using the
.randomSplit()method without shuffling to improve performance. - D
Account for stragglers in distributed data processing, as they might lead to uneven splits.
- E
Rely on deterministic splits even if the data distribution is uneven.
Show answer and explanation
Correct answers: A, B, D
Explanation
When splitting distributed data in Spark ML, it's important to ensure a proper split to avoid data biases and reproducibility issues. Shuffling the data ensures randomness, while using the same random seed guarantees reproducibility. Additionally, in distributed environments, stragglers can impact the balance of the splits and should be accounted for. Avoiding these common pitfalls ensures the training and testing sets are representative and balanced for better model training and evaluation.
- A. Correct.
Shuffling the dataset ensures that the data is randomly distributed before splitting, which reduces the likelihood of biased splits and ensures the training and testing sets are representative of the overall dataset.
- B. Correct.
Using the same random seed ensures that the split is reproducible, which is critical for debugging and comparing results across different runs.
- C. Incorrect.
Not shuffling the dataset before splitting can lead to biased splits, especially if the data is ordered or grouped by certain features. This can negatively affect model performance.
- D. Correct.
In distributed environments, stragglers (slower tasks) can cause uneven processing and splits. Proper monitoring and handling of such cases ensure better-balanced splits.
- E. Incorrect.
Relying purely on deterministic splits without addressing uneven data distribution can lead to imbalanced datasets, where one split may contain significantly more data or diversity than the other.