Databricks Machine Learning Professional Question 199
Select 3You are tasked with building a real-time machine learning pipeline for a stock price prediction application using Databricks. The system must process streaming data from a Kafka topic and make predictions using a pre-trained model deployed on Databricks. What components and configurations would you include to ensure the pipeline meets the real-time requirements?
- A
Use Structured Streaming in Databricks to process data from the Kafka topic.
- B
Deploy the model as a batch inference pipeline and schedule it to run every hour.
- C
Leverage Databricks Auto Loader to automatically manage file-based streaming inputs.
- D
Enable model serving on Databricks to deploy the model as a REST API for real-time predictions.
- E
Use Delta Lake to store streaming data for downstream analysis and checkpointing.
Show answer and explanation
Correct answers: A, D, E
Explanation
To build a real-time machine learning pipeline in Databricks, you need to integrate components that support low-latency streaming data processing and inference. Structured Streaming is essential for consuming data from Kafka in real-time. Model serving as a REST API enables low-latency predictions, while Delta Lake ensures data storage reliability and checkpointing, which is critical for fault-tolerant streaming pipelines. Together, these components address both real-time processing and inference requirements.
- A. Correct.
Correct: Structured Streaming in Databricks is designed for real-time processing of streaming data from sources like Kafka, making it essential for a real-time pipeline.
- B. Incorrect.
Incorrect: Batch inference scheduled to run every hour does not meet the real-time requirements of the application as it introduces significant latency.
- C. Incorrect.
Incorrect: Databricks Auto Loader is useful for file-based streaming inputs but is not relevant for Kafka-based streaming in this scenario.
- D. Correct.
Correct: Enabling model serving on Databricks allows the model to be deployed as a REST API, which supports real-time inference.
- E. Correct.
Correct: Delta Lake is suitable for storing streaming data due to its ACID properties and efficient handling of checkpointing, which ensures fault tolerance in the pipeline.