Databricks Machine Learning Associate Question 638
Select 2You are tasked with performing batch inference on a large dataset using a pre-trained machine learning model in a Databricks notebook. The model expects input features in the form of a pandas DataFrame. Which steps should you include to ensure the batch inference is performed efficiently?
- A
Load the input dataset into a pandas DataFrame and split it into smaller batches for processing.
- B
Use the pandas
apply()function to pass each row of the DataFrame to the model for inference. - C
Leverage vectorized operations or model methods that accept pandas DataFrames to perform inference on entire batches at once.
- D
Save the inference results directly from pandas to a Delta table using the
to_delta()method. - E
Convert the pandas DataFrame into a PySpark DataFrame before performing inference for better scalability.
Show answer and explanation
Correct answers: A, C
Explanation
For efficient batch inference using pandas in Databricks, it is crucial to split large datasets into smaller chunks to avoid memory issues. Additionally, leveraging vectorized operations or model methods that process entire DataFrames at once ensures faster inference compared to row-by-row processing. The other options either involve incorrect pandas methods or unnecessary conversions that are not aligned with the use of pandas for inference.
- A. Correct.
Splitting the dataset into smaller batches ensures efficient processing and avoids memory issues when working with large datasets in pandas.
- B. Incorrect.
Using the pandas
apply()function processes rows one at a time, which is computationally expensive and inefficient for batch inference. - C. Correct.
Vectorized operations or model methods that accept pandas DataFrames allow for efficient batch inference by processing multiple rows simultaneously.
- D. Incorrect.
The
to_delta()method is not a valid pandas method. To save inference results to a Delta table, you would need to convert the pandas DataFrame to a PySpark DataFrame or use other appropriate methods. - E. Incorrect.
While PySpark is suitable for large-scale data processing, converting a pandas DataFrame to a PySpark DataFrame is unnecessary if the inference is specifically designed to work with pandas.