Databricks Machine Learning Professional Question 198
Single answerYou are tasked with building a real-time recommendation system for a video streaming platform using Databricks. The platform requires that recommendations be updated within seconds of user interactions (e.g., likes, views). Which of the following approaches would best meet this requirement?
- A
Use a batch processing pipeline with daily updates to the recommendation model.
- B
Leverage Structured Streaming in Databricks to process user interaction data in near real-time and update recommendations.
- C
Generate recommendations offline and cache them in a Delta table for quick access.
- D
Use Databricks AutoML to train a model and manually re-train it monthly to refresh recommendations.
Show answer and explanation
Correct answer: B
Explanation
Structured Streaming in Databricks is the best approach for real-time or near real-time requirements. It allows continuous data processing, ensuring that recommendations are updated dynamically as user interactions occur. Other approaches either fail to meet the real-time criteria or involve static updates that do not match the required responsiveness.
- A. Incorrect.
Batch processing with daily updates is not suitable for real-time requirements since the data and recommendations would only be refreshed once per day, causing significant delays.
- B. Correct.
Structured Streaming in Databricks is designed for real-time or near real-time data processing. It can continuously process user interaction data and update recommendations dynamically, meeting the requirement for updates within seconds.
- C. Incorrect.
Generating recommendations offline and caching them in a Delta table provides fast access but does not allow for real-time updates based on user interactions.
- D. Incorrect.
While Databricks AutoML can help train models, manually re-training the model monthly does not address the need for real-time updates.