Databricks Machine Learning Professional Question 151
Select 4You have trained a single-node machine learning model locally and now want to deploy it in a distributed manner using Spark on Databricks. To achieve this, you decide to use a Spark UDF (User Defined Function). Which of the following steps are necessary to successfully deploy and use the model in parallel using spark_udf?
- A
Serialize the trained model and store it in a distributed file system such as DBFS.
- B
Define a Python function to load the serialized model and perform predictions.
- C
Register the Python function as a Spark UDF to apply the model in parallel on a Spark DataFrame.
- D
Train the model directly on a Spark DataFrame using the PySpark MLlib API.
- E
Use the
mapPartitionsfunction instead of a UDF since UDFs cannot handle parallel processing. - F
Ensure that the Spark cluster has sufficient resources to handle the model's parallel inference workload.
Show answer and explanation
Correct answers: A, B, C, F
Explanation
To deploy a single-node model in parallel using Spark UDF, the model must first be serialized and stored in a distributed file system like DBFS. A Python function is then defined to deserialize the model and perform predictions. This function is registered as a Spark UDF, allowing it to be applied to the data in parallel across the Spark Distributed DataFrame. Additionally, sufficient cluster resources must be provisioned to handle the inference workload. Options like training the model using PySpark MLlib or using mapPartitions are not relevant to deploying a pre-trained single-node model using Spark UDF.
- A. Correct.
Correct: The trained model must be serialized (e.g., using pickle, joblib, or similar libraries) and stored in a location accessible to all Spark worker nodes, such as DBFS.
- B. Correct.
Correct: A Python function is necessary to deserialize the model and perform predictions when the UDF is called on each partition or row.
- C. Correct.
Correct: Registering the Python function as a Spark UDF is a key step to enable the model to be applied to the data in parallel across the Spark cluster.
- D. Incorrect.
Incorrect: Training the model directly using PySpark MLlib is unrelated to deploying a pre-trained single-node model using Spark UDF.
- E. Incorrect.
Incorrect: While
mapPartitionscan distribute computations, Spark UDFs are specifically designed for row-wise parallel operations and are appropriate for this scenario. - F. Correct.
Correct: Sufficient cluster resources (e.g., memory and CPU) are required to handle the parallel inference workload efficiently when using Spark UDFs.