Databricks Machine Learning Associate Question 283
Single answerYou are working on a machine learning pipeline in Databricks using Spark ML. You need to split a dataset into training and testing sets while ensuring that 80% of the data is used for training and 20% is used for testing. Which of the following code snippets correctly achieves this split in PySpark?
- A
train, test = dataset.randomSplit([0.8, 0.2], seed=42)
- B
train, test = dataset.split([0.8, 0.2], seed=42)
- C
train, test = dataset.randomSplit([0.2, 0.8], seed=42)
- D
train, test = dataset.randomSplit([0.8, 0.2])
Show answer and explanation
Correct answer: A
Explanation
In PySpark's Spark ML, the randomSplit method is used to split datasets into training and testing sets. You provide an array of proportions that specify the fraction of data for each split. The seed is optional but highly recommended to ensure reproducibility of the split, especially in machine learning workflows where consistent results are crucial. The first option correctly uses randomSplit with appropriate proportions (80% training, 20% testing) and a seed for reproducibility.
- A. Correct.
This is the correct syntax for splitting the dataset in Spark ML using the
randomSplitmethod with the correct proportions (80% training and 20% testing) and an optional seed for reproducibility. - B. Incorrect.
The
splitmethod is not a valid function in PySpark for splitting datasets; the correct method israndomSplit. - C. Incorrect.
This uses the wrong proportions, with 20% for training and 80% for testing, which does not meet the requirements.
- D. Incorrect.
While this syntax correctly splits the dataset into 80% training and 20% testing, it does not include a seed, making the results non-reproducible. Including a seed is recommended for reproducibility in experiments.