Databricks Machine Learning Professional Question 192
Select 4You are tasked with converting a batch machine learning inference pipeline into a streaming deployment pipeline on Databricks. The current batch pipeline processes input files stored in a cloud storage location and writes the predictions to another cloud storage location as parquet files. Which of the following steps are required to successfully implement this conversion?
- A
Use a structured streaming read operation to continually consume new input data from the cloud storage location.
- B
Replace the batch-trained model with an online learning model that updates in real-time.
- C
Use a structured streaming write operation to output predictions to the target storage location.
- D
Switch the model prediction logic to process each incoming micro-batch of data.
- E
Enable checkpointing to maintain state and fault tolerance in the streaming pipeline.
Show answer and explanation
Correct answers: A, C, D, E
Explanation
To convert a batch deployment pipeline into a streaming deployment pipeline, the system must transition from processing all data at once to processing data incrementally as it arrives. This involves using structured streaming read and write operations, modifying the model prediction logic for micro-batch processing, and enabling checkpointing for resilience. Replacing the model with an online learning model is unnecessary as batch-trained models can still be used for inference in streaming scenarios.
- A. Correct.
Correct. A structured streaming read operation is necessary to consume new data as it arrives in the storage location instead of processing all data at once as in batch.
- B. Incorrect.
Incorrect. The existing batch-trained model can be reused for inference in a streaming pipeline. Online learning models are not mandatory for streaming inference.
- C. Correct.
Correct. A structured streaming write operation ensures that the output predictions are written incrementally as new data is processed, aligning with the streaming paradigm.
- D. Correct.
Correct. The model prediction logic must be adapted to process each incoming micro-batch, enabling the streaming pipeline to handle data in small, incremental steps.
- E. Correct.
Correct. Enabling checkpointing is essential in streaming pipelines to maintain state (e.g., progress tracking) and ensure fault tolerance in case of failures.