Databricks Machine Learning Associate Question 590
Single answerYou are using Databricks to train a machine learning model for predicting house prices. To ensure reliable model evaluation and prevent overfitting, you decide to use cross-validation. Which of the following statements is true about how cross-validation can be used within Databricks?
- A
Cross-validation splits the dataset into training and testing sets and trains the model on only one fold of the data.
- B
Cross-validation involves splitting the dataset into multiple folds, training the model on a subset of the folds, and validating it on the remaining fold.
- C
Cross-validation trains the model on the entire dataset and then evaluates it on the same dataset for performance metrics.
- D
Cross-validation in Databricks only works with Spark ML models and is not compatible with scikit-learn models.
Show answer and explanation
Correct answer: B
Explanation
Cross-validation is a best practice for model evaluation as it helps assess the model's performance on unseen data. In Databricks, you can implement cross-validation using either Spark ML or libraries like scikit-learn. The process involves splitting the dataset into multiple folds, training the model on some folds, and validating it on the remaining fold. This ensures a robust evaluation and prevents overfitting, providing a more accurate estimate of model performance.
- A. Incorrect.
This is incorrect because cross-validation does not train the model on only one fold of the data; it uses multiple folds to ensure robust evaluation.
- B. Correct.
This is correct because cross-validation involves dividing the dataset into multiple folds, training the model on a subset (e.g., k-1 folds), and validating it on the remaining fold. This process is repeated for all folds, and the results are averaged to evaluate the model's performance.
- C. Incorrect.
This is incorrect because cross-validation does not train and evaluate on the same dataset. Doing so would lead to overfitting and give an overly optimistic performance estimate.
- D. Incorrect.
This is incorrect because Databricks supports cross-validation with both Spark ML and scikit-learn models, as it is a flexible platform suitable for multiple libraries.