Databricks Machine Learning Professional Question 173
Single answerA retail company wants to build a machine learning model to predict fraudulent transactions as they occur in real-time. They are using a Databricks cluster and want to continuously process incoming transactional data from a Kafka stream. Which approach is most suitable for deploying the model inference in this scenario?
- A
Use Databricks Structured Streaming to load the Kafka stream and integrate the model for batch prediction.
- B
Deploy the model as a REST API and invoke it for each incoming transaction in the Kafka stream.
- C
Use Databricks Structured Streaming to load the Kafka stream and perform continuous inference with the model.
- D
Load the Kafka stream into Delta Lake in batch intervals and perform inference using SQL queries.
Show answer and explanation
Correct answer: C
Explanation
Databricks Structured Streaming provides a scalable and fault-tolerant framework for real-time processing of streaming data, such as Kafka streams. By integrating the pre-trained machine learning model with Structured Streaming, the company can perform continuous inference on each incoming transaction without unnecessary delays. This makes Option 3 the most suitable approach for this scenario.
- A. Incorrect.
Structured Streaming can integrate with Kafka, but batch prediction is not continuous and would introduce latency, which is not suitable for real-time inference.
- B. Incorrect.
REST APIs can handle real-time inference but are not as efficient or scalable as Structured Streaming when working with high-throughput data streams directly in Databricks.
- C. Correct.
Structured Streaming is specifically designed for real-time, continuous processing of streaming data, making it the optimal solution for continuous inference in this use case.
- D. Incorrect.
Loading data into Delta Lake in batch intervals does not provide real-time continuous inference as the process involves latency between batch loads.