Databricks Machine Learning Professional Question 190
Select 3You are tasked with converting an existing machine learning batch inference pipeline to a streaming inference pipeline using Databricks. The batch pipeline processes input data stored in a Delta table, applies a machine learning model for prediction, and writes the results back to another Delta table. How should you handle this transition to ensure the pipeline works with real-time streaming data?
- A
Use a Structured Streaming read operation to continuously monitor and load new data into the pipeline.
- B
Replace the batch write operation with a Structured Streaming write that supports the 'append' output mode.
- C
Convert the machine learning model to a real-time compatible API like REST and call it in the streaming pipeline.
- D
Switch the Delta table format to Parquet to optimize for streaming data.
- E
Configure the input stream to use a trigger interval, ensuring a micro-batch processing approach.
Show answer and explanation
Correct answers: A, B, E
Explanation
To convert a batch inference pipeline into a streaming pipeline, the input and output operations must be adapted to work with Structured Streaming. This ensures data is continuously read, processed, and written in real-time. Additionally, configuring a trigger interval allows the pipeline to process data in manageable micro-batches, maintaining efficiency and responsiveness.
- A. Correct.
Structured Streaming is required to enable the pipeline to handle real-time data by continuously monitoring and loading new data. Without this, the pipeline will not operate in a streaming mode.
- B. Correct.
For streaming pipelines, the write operation must be adapted to support streaming output modes like 'append' to ensure the predictions are written incrementally.
- C. Incorrect.
While using a REST API for real-time model inference is possible, it is not necessary in this case as the pipeline already uses a model for inference. The pipeline can directly apply the model to streaming data.
- D. Incorrect.
Switching to Parquet format does not improve streaming compatibility, as Delta tables are already optimized for both batch and streaming use cases.
- E. Correct.
Using a trigger interval is important for controlling the frequency of micro-batches in Structured Streaming, making the pipeline efficient and compatible with streaming data.