Databricks Machine Learning Associate Question 281
Single answerYou are working on a machine learning pipeline in Databricks and need to split your dataset into training and testing sets using Spark ML. Which of the following code snippets correctly splits the data into a 70% training set and a 30% testing set?
- A
train_data, test_data = df.randomSplit([0.7, 0.3], seed=42)
- B
train_data, test_data = df.split([0.7, 0.3], seed=42)
- C
train_data, test_data = df.randomSplit([0.3, 0.7], seed=42)
- D
train_data, test_data = df.randomSplit([0.7, 0.3])
Show answer and explanation
Correct answer: A
Explanation
The randomSplit method in Spark ML is used to split datasets into multiple subsets based on the specified proportions. For a 70-30 split, you need to provide [0.7, 0.3] as the first argument. Including a seed ensures that the split is reproducible, which is a best practice in machine learning workflows.
- A. Correct.
This is the correct code to split a dataset into training and testing sets using Spark ML. The
randomSplitmethod takes a list of split proportions as its first argument and an optionalseedargument for reproducibility. - B. Incorrect.
This is incorrect because
splitis not a valid method in Spark ML for splitting datasets. The correct method israndomSplit. - C. Incorrect.
This is incorrect because the proportions for the training and testing sets are reversed. A 70% training set and 30% testing set require the proportions
[0.7, 0.3]. - D. Incorrect.
This is incorrect because while the proportions are correct, the absence of a
seedargument makes the split non-deterministic, which is not advisable in a professional or reproducible setting.