Databricks Machine Learning Associate Question 636
Select 4You have trained a machine learning model in Databricks and saved it as a .pkl file. You now want to perform batch inference on a new dataset stored as a CSV file using pandas in a Databricks notebook. Which of the following steps are required to correctly perform batch inference in this scenario?
- A
Load the new dataset using pandas'
read_csvmethod. - B
Load the trained model using the
pickle.loadfunction. - C
Use the
model.predictmethod on the entire pandas DataFrame without any preprocessing. - D
Ensure the feature columns in the new dataset match those used during training.
- E
Save the inference results back to the Databricks file system using pandas'
to_csvmethod.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Performing batch inference with pandas in Databricks involves loading the new dataset, ensuring it is preprocessed to match the training dataset's feature structure, loading the trained model, applying the model's predict method, and finally saving the results. Skipping preprocessing could lead to errors or inaccurate predictions, which is why preprocessing must be considered carefully.
- A. Correct.
Correct: The new dataset must be loaded into a pandas DataFrame using
read_csvbefore performing batch inference. - B. Correct.
Correct: The trained model, saved as a
.pklfile, needs to be loaded usingpickle.loadto use it for inference. - C. Incorrect.
Incorrect: The new dataset might require preprocessing (e.g., scaling, encoding) to match the format expected by the model. Simply calling
model.predictwithout preprocessing can lead to errors. - D. Correct.
Correct: The feature columns in the new dataset must match the structure and order of the dataset used during training to ensure accurate predictions.
- E. Correct.
Correct: After performing batch inference, saving the results back to the Databricks file system using pandas'
to_csvis a common step for further analysis or sharing.