Databricks Machine Learning Professional Question 166
Select 3You are building a machine learning pipeline in Databricks to process streaming data from a Kafka source. The pipeline requires real-time predictions from a trained machine learning model. Which of the following steps are necessary to correctly implement this streaming pipeline in Databricks?
- A
Use Spark Structured Streaming to read data from the Kafka source.
- B
Deploy the trained model as a REST API and make predictions by calling the API within the streaming query.
- C
Use the
foreachBatchoperation to apply the trained model on micro-batches of the streaming data. - D
Write the streaming query results directly to a feature store for long-term storage.
- E
Ensure the streaming query has a checkpoint location configured to maintain state.
Show answer and explanation
Correct answers: A, C, E
Explanation
To implement a streaming pipeline in Databricks for real-time machine learning predictions, you need to use Spark Structured Streaming to read data from the Kafka source. The trained model can be applied using the foreachBatch operation, which processes micro-batches of the streaming data directly within Spark. Additionally, you must configure a checkpoint location to maintain state and ensure fault tolerance. Deploying the model as a REST API or writing results to a feature store might be useful for other use cases but are not essential for this specific scenario.
- A. Correct.
Correct: Spark Structured Streaming is required to read data from streaming sources like Kafka in Databricks.
- B. Incorrect.
Incorrect: While deploying the model as a REST API is a valid approach in some cases, it is not efficient for real-time predictions within a Databricks streaming pipeline. The model should be used directly within the Spark environment to minimize latency.
- C. Correct.
Correct: The
foreachBatchoperation allows you to process micro-batches, making it suitable for applying a trained machine learning model to streaming data. - D. Incorrect.
Incorrect: While writing results to a feature store is useful for offline use cases, it is not a required step for real-time streaming pipelines.
- E. Correct.
Correct: Configuring a checkpoint location is essential for maintaining state in a streaming query and ensuring exactly-once processing semantics in Databricks.