Databricks Machine Learning Professional Question 189
Select 3You have developed a batch inference pipeline using Databricks that processes a large dataset stored in a Delta table and outputs predictions back into another Delta table. Your team now requires the pipeline to handle real-time streaming data from a Kafka source instead. Which of the following steps are necessary to convert the batch pipeline into a streaming pipeline in Databricks?
- A
Replace the batch read operation with a streaming read from Kafka using the
readStreammethod. - B
Modify the model's inference logic to handle one record at a time instead of batch processing.
- C
Replace the batch write operation with a streaming write using the
writeStreammethod. - D
Enable time windowing in the streaming pipeline to handle data arriving late.
- E
Use a trigger interval to control the frequency of processing in the streaming pipeline.
Show answer and explanation
Correct answers: A, C, E
Explanation
To convert a batch deployment pipeline to a streaming deployment pipeline in Databricks, you need to replace batch operations with their streaming equivalents, such as readStream and writeStream, and configure a trigger interval to define the processing frequency. Modifying the model's inference logic or enabling time windowing is not always required unless the use case specifically demands it.
- A. Correct.
Correct: To handle real-time data, you need to use a streaming source like Kafka and replace the batch
readoperation with a streamingreadStreamoperation. - B. Incorrect.
Incorrect: The model's inference logic can often remain unchanged as Spark processes micro-batches in streaming mode, so there is no requirement to modify it to handle one record at a time.
- C. Correct.
Correct: In a streaming pipeline, you must replace batch write operations with a streaming write operation using the
writeStreammethod. - D. Incorrect.
Incorrect: While time windowing is useful for certain use cases like aggregating data, it is not a mandatory step for converting a batch pipeline to a streaming pipeline.
- E. Correct.
Correct: Using a trigger interval is an important step to control how frequently the streaming pipeline processes incoming data.