Databricks Machine Learning Associate Question 289
Select 3You are tasked with building and evaluating a machine learning model using Spark ML in Databricks. After splitting your dataset into training and testing sets, you use the LogisticRegression model from Spark MLlib. Which of the following steps are necessary to properly train and evaluate the model?
- A
Fit the
LogisticRegressionmodel on the training data using the.fit()method. - B
Use the
.transform()method on the testing data to generate predictions. - C
Directly use the
.fit()method on the entire dataset to generate predictions. - D
Evaluate the model's predictions using an appropriate metric, such as accuracy or area under the ROC curve.
- E
Convert the Spark DataFrame into a Pandas DataFrame before fitting the model.
Show answer and explanation
Correct answers: A, B, D
Explanation
When training and evaluating a machine learning model using Spark ML, it is essential to follow the proper workflow: split the dataset, train the model using .fit() on the training data, use .transform() on the testing data to generate predictions, and evaluate the predictions using suitable metrics. Spark ML models operate directly on Spark DataFrames, so conversion to Pandas DataFrames is not required.
- A. Correct.
This is correct. The
.fit()method is used to train the model on the training data in Spark ML. - B. Correct.
This is correct. The
.transform()method is used to apply the trained model to the testing data and generate predictions. - C. Incorrect.
This is incorrect. Training on the entire dataset without splitting into training and testing sets violates the principle of model evaluation, as it would lead to overfitting and unreliable performance metrics.
- D. Correct.
This is correct. Evaluating the model's predictions using appropriate metrics ensures that you can assess its performance effectively.
- E. Incorrect.
This is incorrect. While Spark DataFrames can be converted to Pandas DataFrames, this step is unnecessary for training and evaluating Spark ML models.