Databricks Machine Learning Professional Question 196
Select 3You are tasked with building a real-time machine learning pipeline for fraud detection using Databricks. The pipeline must process streaming transaction data, score each transaction in real-time, and alert the system if fraud is detected. Which combination of Databricks tools and frameworks should you use to achieve this?
- A
Databricks Structured Streaming for processing the streaming data
- B
MLflow Model Registry to manage and deploy the fraud detection model
- C
Databricks Auto Loader for loading historical batch data
- D
Delta Lake to store the transaction data and maintain a reliable state
- E
Apache Spark MLlib for building the fraud detection model
Show answer and explanation
Correct answers: A, B, D
Explanation
To build a real-time machine learning pipeline on Databricks, you need tools for streaming data processing (Structured Streaming), model management and deployment (MLflow Model Registry), and reliable state management (Delta Lake). Structured Streaming handles live data ingestion, MLflow ensures scalable model deployment, and Delta Lake provides a reliable data layer. Auto Loader and Spark MLlib, while useful in other contexts, are not directly required for this real-time use case.
- A. Correct.
Databricks Structured Streaming is designed for processing streaming data in real-time, making it a crucial component of a real-time pipeline.
- B. Correct.
MLflow Model Registry enables model versioning and deployment, which is essential for deploying the fraud detection model in real-time.
- C. Incorrect.
Databricks Auto Loader is useful for loading historical batch data but is not directly relevant to the processing of real-time streaming data.
- D. Correct.
Delta Lake provides ACID transactions and a reliable storage layer, which is essential for managing the state of the system and transaction data in real-time pipelines.
- E. Incorrect.
Apache Spark MLlib is useful for building models, but it is not specifically required for managing or deploying a real-time pipeline. Pre-trained models can also be deployed without using MLlib.