Databricks Machine Learning Associate Question 561
Select 3You are tasked with developing a machine learning model to predict customer churn. The model will be trained using a Databricks notebook, and the training pipeline needs to be modular, reusable, and capable of running on new data without manual intervention. Which of the following steps should you include in your training pipeline?
- A
Load the data from a Delta Lake table into a Spark DataFrame to ensure data versioning and consistency.
- B
Perform data preprocessing and feature engineering steps such as handling missing values and scaling features directly within the model training function.
- C
Split the data into training and test datasets and log the split ratios in MLflow for reproducibility.
- D
Train the model using the training dataset and log the model parameters, metrics, and artifacts in MLflow.
- E
Manually evaluate the model performance on the test dataset and save the evaluation results to a local file for later review.
Show answer and explanation
Correct answers: A, C, D
Explanation
Creating a robust training pipeline in Databricks involves ensuring modularity, automation, and reproducibility. Loading data from Delta Lake ensures data consistency, while splitting data and logging details in MLflow enhances reproducibility. Logging parameters, metrics, and artifacts in MLflow ensures traceability. Efficient training pipelines avoid manual steps and separate preprocessing from model training to ensure modularity and reusability.
- A. Correct.
Loading data from a Delta Lake table ensures data consistency, versioning, and reliability, which are essential for a robust training pipeline.
- B. Incorrect.
Performing preprocessing and feature engineering within the model training function is not a modular practice. These steps should be separate to ensure reusability and clarity of the pipeline.
- C. Correct.
Splitting the data and logging the split ratios in MLflow helps with reproducibility, which is a critical requirement for a training pipeline.
- D. Correct.
Logging model parameters, metrics, and artifacts in MLflow ensures traceability and allows other users to understand and reuse the model effectively.
- E. Incorrect.
Manually evaluating and saving results to a local file is not aligned with the principles of automation and reproducibility required for a robust training pipeline.