Databricks Machine Learning Associate Question 280
Single answerYou are working on a machine learning pipeline in Databricks using Spark MLlib. You need to split your dataset into training and test sets to evaluate your model. Which of the following code snippets correctly splits the dataset into 80% training data and 20% test data?
- A
trainingData, testData = dataset.randomSplit([0.8, 0.2])
- B
trainingData, testData = dataset.split([0.8, 0.2])
- C
trainingData, testData = dataset.randomSplit([0.8, 0.2], seed=42)
- D
trainingData, testData = dataset.randomSplit([80, 20])
Show answer and explanation
Correct answer: C
Explanation
In Spark MLlib, the 'randomSplit' method is used to split a dataset into multiple subsets based on the specified proportions. Including a random seed ensures the split is reproducible. Proportions must be provided as decimal values that sum up to 1, and specifying a seed is a best practice for consistent results.
- A. Incorrect.
This code is close to correct, but it does not specify a random seed. Without a seed, the split may lead to inconsistent results when re-executed.
- B. Incorrect.
This code is incorrect because the 'split' method does not exist in Spark MLlib. The correct method is 'randomSplit'.
- C. Correct.
This is the correct code for splitting the dataset into 80% training and 20% test data. It uses 'randomSplit' with proportions and includes a random seed for reproducibility.
- D. Incorrect.
This code is incorrect because the proportions for 'randomSplit' must be specified as decimals (e.g., 0.8 and 0.2), not as integers (e.g., 80 and 20).