Databricks Machine Learning Professional Question 133
Select 3You are tasked with deploying a machine learning model in Databricks to generate predictions for a large dataset in a batch process. You need to ensure the predictions are saved for later use. Which of the following steps best describe how you can achieve this?
- A
Load the input dataset into a Spark DataFrame and apply the model's transformation function to compute predictions.
- B
Write the predictions to a persistent storage location such as a Delta table or an external cloud storage bucket.
- C
Use Databricks REST API to directly serve real-time predictions and store them in memory for later.
- D
Schedule the batch prediction job using a Databricks job or workflow to automate the process.
- E
Deploy the model as a REST endpoint and query it repeatedly for batch predictions.
Show answer and explanation
Correct answers: A, B, D
Explanation
Batch deployment in Databricks involves loading the input data into a Spark DataFrame, applying the model to compute predictions, and saving the predictions to a persistent storage location such as a Delta table or cloud storage. To ensure automation and scalability, the process is often scheduled using Databricks jobs or workflows. Real-time APIs and REST endpoints are not suitable for batch processes, as they are designed for real-time inference.
- A. Correct.
Correct: In a batch deployment, the model is applied to a dataset (such as a Spark DataFrame) to compute predictions.
- B. Correct.
Correct: The predictions need to be saved to persistent storage for later use, such as a Delta table or cloud storage.
- C. Incorrect.
Incorrect: The REST API is typically used for real-time predictions, not batch processing. Additionally, storing predictions in memory is not suitable for large datasets.
- D. Correct.
Correct: Automating the batch prediction process using a Databricks job or workflow ensures predictions are computed and saved on a schedule or as needed.
- E. Incorrect.
Incorrect: Deploying the model as a REST endpoint is not efficient or practical for batch predictions, as it is designed for real-time use cases.