Databricks Machine Learning Professional Question 188
Select 4You are tasked with converting a batch inference pipeline into a streaming inference pipeline for a machine learning model deployed on Databricks. The batch pipeline currently reads data from a Delta table, applies a trained model for inference, and writes the predictions back to a different Delta table. What steps should you take to make this pipeline work in a streaming context?
- A
Use a Delta table with 'readStream()' to continuously read new incoming data.
- B
Replace the batch prediction function with a streaming-compatible function that processes data incrementally.
- C
Enable Structured Streaming by using 'writeStream()' to output results to a sink.
- D
Switch from Delta tables to CSV files for both input and output data to support streaming.
- E
Use a checkpoint location for the streaming pipeline to ensure fault tolerance and exactly-once processing.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
Converting a batch inference pipeline to a streaming deployment pipeline requires key changes to enable real-time data processing. Using 'readStream()' and 'writeStream()' ensures the pipeline can read and write data continuously. A streaming-compatible prediction function processes data incrementally, and checkpointing preserves state and guarantees fault tolerance. Delta tables are preferred for streaming due to their support for ACID transactions and scalability.
- A. Correct.
Correct: 'readStream()' is required to enable streaming reads from a Delta table or other supported data sources.
- B. Correct.
Correct: Streaming inference requires processing data incrementally, so the batch prediction function must be adapted to handle data one micro-batch at a time.
- C. Correct.
Correct: 'writeStream()' is necessary to output the streaming predictions to a supported sink, such as a Delta table or console.
- D. Incorrect.
Incorrect: While CSV files can be used as a data source/sink, they do not inherently support streaming in the same way Delta tables do, leading to potential inefficiencies or data loss.
- E. Correct.
Correct: Checkpointing is critical for maintaining state and ensuring fault tolerance in a structured streaming pipeline.