Databricks Machine Learning Professional Question 194
Select 4You are tasked with deploying a Databricks machine learning model for real-time predictions. The current batch deployment pipeline reads data from a Delta table, processes it, and writes predictions back to a new Delta table. You need to convert this batch deployment pipeline into a streaming deployment pipeline to handle real-time data ingestion and prediction. Which of the following steps are necessary to achieve this?
- A
Use the
readStreammethod instead ofreadto read the input data as a streaming source. - B
Replace the batch-trained model with a real-time trained model specifically designed for streaming.
- C
Ensure the model inference logic can handle micro-batch processing by integrating it into the
foreachBatchmethod. - D
Use the
writeStreammethod to write the predictions back to the Delta table or another supported sink. - E
Configure a trigger interval in the streaming pipeline to control the frequency of prediction batch processing.
Show answer and explanation
Correct answers: A, C, D, E
Explanation
To convert a batch deployment pipeline to a streaming deployment pipeline in Databricks, you need to enable streaming using readStream and writeStream. The model inference logic must be adapted to process micro-batches, usually by using the foreachBatch method. Additionally, configuring a trigger interval ensures the pipeline processes data at the desired frequency. It is not necessary to retrain the model specifically for streaming, as most batch-trained models can be used for inference in structured streaming scenarios.
- A. Correct.
Correct. Using the
readStreammethod is essential to enable the pipeline to process data in real-time instead of batch mode. - B. Incorrect.
Incorrect. While the model should be capable of handling streaming data, it does not necessarily need to be specifically retrained for real-time processing. Most batch-trained models can still be used for streaming inference.
- C. Correct.
Correct. The
foreachBatchmethod allows you to apply custom logic, including model inference, to each micro-batch of data in a structured streaming pipeline. - D. Correct.
Correct. The
writeStreammethod is required to output the predictions in a streaming fashion to a supported sink such as a Delta table, Kafka, or a file system. - E. Correct.
Correct. Configuring a trigger interval, such as
trigger(once=True)or a custom interval, helps control the frequency of data processing and prediction generation in the streaming pipeline.