Databricks Machine Learning Professional Question 149
Select 3You have trained a machine learning model on a single machine and now need to deploy it to process predictions in parallel using Apache Spark in a Databricks environment. The goal is to use a Spark UDF to apply the model in parallel across a large dataset stored in a DataFrame. Which of the following steps are required to achieve this?
- A
Serialize the trained model into a format supported by the Spark UDF (e.g., using joblib or pickle).
- B
Write a Python function that loads the serialized model and performs predictions, and then register it as a Spark UDF.
- C
Directly pass the trained model object to the Spark UDF without serialization.
- D
Apply the Spark UDF to a Spark DataFrame column to perform predictions in parallel.
- E
Ensure the model is retrained on the entire Spark cluster before using a Spark UDF for predictions.
Show answer and explanation
Correct answers: A, B, D
Explanation
Deploying a single-node model using a Spark UDF involves serializing the trained model, writing a function to load the model and perform predictions, and registering this function as a UDF. The UDF can then be applied to a Spark DataFrame for parallel processing. Serialization and proper UDF registration are essential for ensuring the model is distributed correctly to worker nodes.
- A. Correct.
Correct: The trained model must be serialized into a format that can be loaded within the Spark UDF on each worker node. This ensures the model is available for distributed processing.
- B. Correct.
Correct: A Python function is needed to load the serialized model and perform predictions. Registering this function as a Spark UDF allows it to be applied to the DataFrame in a distributed manner.
- C. Incorrect.
Incorrect: Directly passing the trained model object to the Spark UDF is not feasible because Spark distributes work across worker nodes, and each node requires its own instance of the model, typically loaded from a serialized file.
- D. Correct.
Correct: Applying the Spark UDF to a DataFrame column enables parallel predictions across the data in the Spark cluster.
- E. Incorrect.
Incorrect: Retraining the model on the entire Spark cluster is not necessary for this scenario. The model can be trained on a single machine and used for inference in a distributed fashion.