Databricks Machine Learning Professional Question 195
Select 4You have developed a machine learning model to predict product demand using a batch inference pipeline that processes daily sales data. Your team now requires real-time predictions as new sales transactions are recorded in the system. Which of the following steps are necessary to convert the batch deployment pipeline into a streaming deployment pipeline in Databricks?
- A
Use a streaming source to read incoming data in real-time, such as Kafka or Delta table with append-only updates.
- B
Replace batch data transformations with Structured Streaming transformations to support incremental data processing.
- C
Continue using batch prediction APIs, as they are compatible with streaming data sources.
- D
Configure a trigger for the streaming query to define the frequency of micro-batch processing.
- E
Persist streaming predictions to a sink, such as a Delta table, to ensure predictions are stored incrementally.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
To transition from a batch deployment pipeline to a streaming deployment pipeline in Databricks, you need to use streaming data sources and Structured Streaming transformations to handle real-time data processing. Batch APIs and transformations are incompatible with streaming pipelines. Additionally, configuring triggers ensures timely processing of data, and persisting the predictions to a sink is essential for storing results of the streaming predictions.
- A. Correct.
Correct: A streaming source (e.g., Kafka or an append-only Delta table) is required to read new incoming data in real-time for a streaming pipeline.
- B. Correct.
Correct: Batch transformations must be replaced with Structured Streaming transformations to process data incrementally as it arrives.
- C. Incorrect.
Incorrect: Batch prediction APIs are not designed for streaming data and would not support real-time processing. Streaming-specific APIs or libraries are needed.
- D. Correct.
Correct: A trigger defines how often the system processes micro-batches in Structured Streaming, enabling real-time or near-real-time predictions.
- E. Correct.
Correct: Streaming predictions must be written to a sink (e.g., a Delta table or other storage) incrementally to ensure predictions are stored correctly as new data arrives.