Databricks Machine Learning Professional Question 164
Single answerYou are working on a machine learning application that processes real-time data using Structured Streaming in Databricks. The goal is to train a classification model that continuously updates every 10 minutes with new incoming data. Which approach should you use to ensure the model is retrained efficiently with the latest data while minimizing resource consumption?
- A
Use a micro-batch processing model to retrain the model every 10 minutes by reading the complete historical dataset.
- B
Use Structured Streaming to read the real-time data and perform incremental model retraining every 10 minutes with only the newly arrived data.
- C
Continuously append the incoming data to a Delta table and retrain the model on a daily basis using the entire dataset.
- D
Use Structured Streaming to read the real-time data and retrain the model for every new record individually as soon as it arrives.
Show answer and explanation
Correct answer: B
Explanation
The correct method for handling real-time data streams in Databricks while building machine learning models is to use Structured Streaming with incremental processing. This allows retraining the model using only the newly arrived data at regular intervals, such as every 10 minutes, reducing resource consumption and ensuring the model stays up-to-date. Other approaches either fail to meet the timing requirement or use computational resources inefficiently.
- A. Incorrect.
This approach is inefficient because reading the complete historical dataset every 10 minutes is computationally expensive and does not leverage the incremental nature of streaming data.
- B. Correct.
This is the correct approach as it allows efficient incremental retraining by processing only newly arrived data, minimizing resource consumption while staying up-to-date with real-time data.
- C. Incorrect.
This approach is suboptimal as it does not meet the requirement to update the model every 10 minutes and delays the retraining process until the end of the day.
- D. Incorrect.
Retraining the model for every new record individually is computationally expensive and unnecessary for most classification tasks, making this approach inefficient.