Databricks Machine Learning Professional Question 191
Select 4You are tasked with converting a batch inference pipeline into a streaming inference pipeline in Databricks to handle real-time data. The existing batch pipeline uses a trained machine learning model to process data stored in a Delta table. What steps should you take to successfully implement this conversion?
- A
Use Spark Structured Streaming to read the data as a stream from the Delta table instead of performing a batch read.
- B
Modify the batch scoring function to process micro-batches of data and return the results for each micro-batch.
- C
Persist the streaming results to a new Delta table configured for append mode.
- D
Switch from Spark Structured Streaming to PySpark RDDs for real-time data processing.
- E
Implement a checkpointing mechanism to ensure fault-tolerant stream processing.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
To convert a batch pipeline to a streaming pipeline in Databricks, Spark Structured Streaming should be utilized to process data streams instead of batch reads. The scoring function must be adapted to process data in micro-batches, and results should be persisted in a Delta table using append mode. Additionally, a checkpointing mechanism must be implemented to ensure fault-tolerant stream processing. Using PySpark RDDs is not recommended for this use case, as Spark Structured Streaming is more suitable for managing streaming workloads efficiently.
- A. Correct.
Correct: Spark Structured Streaming is designed for real-time data processing and supports reading data as a stream from Delta tables, making it essential when converting batch pipelines to streaming.
- B. Correct.
Correct: Batch scoring functions must be updated to handle micro-batches since streaming data arrives in small, continuous chunks rather than all at once.
- C. Correct.
Correct: Streaming results need to be written to a persistent storage like a Delta table for downstream use or analysis, and append mode is typically used in such cases.
- D. Incorrect.
Incorrect: While PySpark RDDs support real-time processing, Spark Structured Streaming is the recommended approach for stream processing in Databricks due to its higher-level API and built-in optimizations for streaming workloads.
- E. Correct.
Correct: Checkpointing is critical for ensuring fault tolerance in stream processing, allowing the pipeline to recover from failures without data loss.