Databricks Machine Learning Associate Question 367
Select 3You are working with a large dataset in a Databricks notebook and need to apply a pre-trained machine learning model to make predictions on a Spark DataFrame. Due to the large size of the data, you decide to use a Pandas UDF. Which of the following steps are necessary to correctly apply the model in parallel using a Pandas UDF?
- A
Define the Pandas UDF with the @pandas_udf decorator and specify the return type.
- B
Ensure the pre-trained model is serialized and broadcast to all worker nodes.
- C
Convert the Spark DataFrame to a Pandas DataFrame using the .toPandas() method before applying the Pandas UDF.
- D
Pass the model predictions as a Pandas Series from the UDF.
- E
Use the Pandas UDF within a .map() transformation on the Spark DataFrame.
Show answer and explanation
Correct answers: A, B, D
Explanation
To apply a model in parallel using a Pandas UDF, you need to ensure the model is available on all worker nodes (via serialization and broadcasting), define the Pandas UDF properly with the @pandas_udf decorator, and return predictions as a Pandas Series. These steps ensure the model is applied efficiently and Spark can process the results. Avoid converting the Spark DataFrame to a Pandas DataFrame, as it can cause memory issues and is not suited for distributed processing. Also, Pandas UDFs are used with DataFrame transformations, not RDD methods like .map().
- A. Correct.
Correct: To define a Pandas UDF in PySpark, you need to use the @pandas_udf decorator and specify the return type so that Spark knows how to handle the resulting data.
- B. Correct.
Correct: When using a pre-trained model in a distributed Spark environment, you need to broadcast the model to worker nodes to ensure it is available for parallel processing.
- C. Incorrect.
Incorrect: Converting the Spark DataFrame to a Pandas DataFrame using .toPandas() would load all data into the driver’s memory, which defeats the purpose of distributed processing and is not scalable.
- D. Correct.
Correct: The Pandas UDF should return predictions as a Pandas Series, which Spark can then combine into the resulting DataFrame.
- E. Incorrect.
Incorrect: Pandas UDFs are used with transformations like .withColumn() or .select(), not with .map(). The .map() transformation is for RDDs, not DataFrames.