Databricks Machine Learning Professional Question 176
Single answerA company is deploying a real-time fraud detection system using Databricks Structured Streaming. The system processes thousands of financial transactions per second and applies machine learning models to detect anomalies. During the development phase, they realize that complex business logic, such as threshold-based rules and contextual checks (e.g., user spending history), needs to be integrated into the streaming pipeline. Why is it important to handle such complex business logic properly in streaming deployments?
- A
Complex business logic ensures consistent model versioning in the pipeline.
- B
Complex business logic helps minimize latency while processing real-time data streams.
- C
Complex business logic ensures the scalability of the streaming application under high load.
- D
Complex business logic ensures accurate and contextual decision-making in real-time.
Show answer and explanation
Correct answer: D
Explanation
In streaming deployments, complex business logic is critical for ensuring that real-time decisions are both accurate and contextually relevant. For example, in fraud detection, ignoring contextual factors like user history or transaction thresholds could lead to false positives or negatives. By properly integrating business logic, the system can provide meaningful and actionable results while processing real-time data streams.
- A. Incorrect.
This is incorrect because consistent model versioning is handled through model registry and deployment strategies, not by business logic integration.
- B. Incorrect.
This is incorrect because adding complex business logic can sometimes increase latency, and minimizing latency usually requires optimizing the logic, not just integrating it.
- C. Incorrect.
This is incorrect because the scalability of a streaming application depends on the underlying infrastructure and proper optimization, not necessarily on the business logic itself.
- D. Correct.
This is correct because integrating complex business logic into the streaming pipeline ensures that decisions, such as fraud detection, are made accurately and in context (e.g., considering user spending history) in real-time.