Databricks Machine Learning Professional Question 167
Single answerYou are tasked with building a machine learning pipeline on Databricks to predict fraudulent transactions in real-time from a stream of financial transaction data. The data is ingested from a Kafka topic and processed using a Delta Live Tables pipeline. As part of the pipeline, you need to ensure that the model is updated with the latest data every hour without interrupting the streaming process. Which approach should you use?
- A
Train a new model using batch data at a fixed interval and deploy it to the streaming job.
- B
Use Databricks AutoML to automatically retrain the model in the streaming pipeline.
- C
Implement model checkpointing to save the model state and update it incrementally without restarting the stream.
- D
Deploy a model serving endpoint and periodically update it with a newly trained model outside the streaming pipeline.
Show answer and explanation
Correct answer: D
Explanation
The best approach to update a machine learning model in a streaming context is to decouple the model retraining and deployment from the streaming process. By using a model serving endpoint, the model can be updated periodically with new training data without interrupting the streaming pipeline. This ensures that the streaming pipeline continues processing data seamlessly while still providing predictions based on the latest model.
- A. Incorrect.
Training a new model in batch and deploying it directly to the streaming job can cause interruptions and is not an efficient way to handle real-time updates.
- B. Incorrect.
Databricks AutoML is not designed to perform real-time or streaming-based training; it is primarily used for batch workflows.
- C. Incorrect.
Model checkpointing is a technique for saving the state of a stream, not for updating machine learning models. It does not solve the problem of updating the model with new data.
- D. Correct.
Deploying a model serving endpoint and updating it periodically outside the streaming pipeline allows the streaming process to remain uninterrupted. The serving endpoint can be updated with a new model trained on the latest batch of data without affecting the ongoing stream.