Databricks Machine Learning Professional Question 148
Select 3You have trained a machine learning model locally and now want to deploy it using a single-node setup in Databricks for batch inference on a large dataset. To execute the inference in parallel using spark_udf, which of the following steps should you take?
- A
Serialize the trained model and load it inside the UDF function.
- B
Use Spark's broadcast variable to distribute the model across the cluster.
- C
Define a Python function that makes predictions using the loaded model.
- D
Apply the UDF on a Spark DataFrame to parallelize the predictions.
- E
Directly call the model's predict function on the Spark DataFrame without using a UDF.
Show answer and explanation
Correct answers: A, C, D
Explanation
To deploy a single-node machine learning model in parallel using spark_udf, the model must first be serialized and loaded inside the UDF to ensure it can be used by Spark workers. A Python function is then defined to encapsulate the prediction logic, and the UDF is applied to the Spark DataFrame to enable distributed inference. Broadcast variables are not necessary for this workflow, and directly calling the model's predict function on a Spark DataFrame is not possible due to Spark's distributed architecture.
- A. Correct.
Correct. The model must be serialized (e.g., using pickle or joblib) and loaded inside the UDF because the UDF will run on Spark workers.
- B. Incorrect.
Incorrect. Spark broadcast variables are not typically used for distributing models in this scenario, as the model is loaded directly within the UDF.
- C. Correct.
Correct. A Python function is required to define the logic for generating predictions using the loaded model.
- D. Correct.
Correct. The UDF is applied to a Spark DataFrame to enable parallelized inference across the dataset.
- E. Incorrect.
Incorrect. The model's predict function cannot be directly called on the Spark DataFrame because Spark operations require distributed processing, which is handled via UDFs.