Databricks Machine Learning Associate Question 364
Select 4You are working with a large dataset in Databricks and need to apply a pre-trained machine learning model to a Spark DataFrame for inference. To parallelize the model inference across multiple rows, you decide to use a Pandas UDF. Which of the following steps are required to successfully implement this approach?
- A
Define the Pandas UDF using the
@pandas_udfdecorator with the appropriate return type. - B
Ensure the model is serialized and broadcasted to the worker nodes before applying the Pandas UDF.
- C
Specify the Pandas UDF as a row-at-a-time function to maximize parallelism.
- D
Pass the Spark DataFrame column(s) as arguments to the Pandas UDF for batch processing.
- E
Load the pre-trained model inside the Pandas UDF to ensure it is available on each worker node.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
To apply a model in parallel using a Pandas UDF, you need to define the UDF with the @pandas_udf decorator and specify the return type. The pre-trained model must be serialized and broadcasted to worker nodes, and the UDF should process data in batches, not row-by-row. Additionally, the model needs to be loaded inside the UDF to make it accessible during execution on the distributed workers.
- A. Correct.
Correct: The
@pandas_udfdecorator is necessary to define a Pandas UDF and specify the return type, such as DoubleType or ArrayType, to ensure Spark understands the output format. - B. Correct.
Correct: Broadcasting the model ensures it is efficiently distributed to all worker nodes, avoiding unnecessary overhead and enabling parallel inference.
- C. Incorrect.
Incorrect: Pandas UDFs operate on batches of data, not row-by-row. Row-at-a-time processing is not compatible with Pandas UDFs and would reduce performance.
- D. Correct.
Correct: For batch processing, the Pandas UDF must accept Spark DataFrame columns as arguments to operate on chunks of data.
- E. Correct.
Correct: Loading the model inside the Pandas UDF ensures that it is available on each worker node during execution, as each worker runs independently.