Databricks Machine Learning Associate Question 562
Select 3You are tasked with building a training pipeline for a machine learning model in Databricks. The pipeline should include data preprocessing, model training, hyperparameter tuning, and logging of model metrics. Which of the following steps are critical to ensure the pipeline is reusable and scalable?
- A
Using Delta Lake for version-controlled input data
- B
Hardcoding hyperparameter values directly in the training script
- C
Logging model metrics and parameters to MLflow
- D
Scheduling the pipeline as a Databricks Job
- E
Writing the training pipeline logic in a single notebook without modularization
Show answer and explanation
Correct answers: A, C, D
Explanation
To ensure that a training pipeline is reusable and scalable, it is important to follow best practices such as using Delta Lake for data versioning, logging metrics to MLflow for tracking experiments, and automating the pipeline through Databricks Jobs. Avoiding hardcoding of hyperparameters and ensuring modularity in the pipeline design are also essential for flexibility and maintainability.
- A. Correct.
Using Delta Lake for input data ensures version control and data consistency, which is essential for reusable and scalable pipelines.
- B. Incorrect.
Hardcoding hyperparameter values directly in the training script makes the pipeline inflexible and difficult to reuse or tune, so this is not recommended.
- C. Correct.
Logging model metrics and parameters to MLflow is critical for tracking experiments and improving reproducibility.
- D. Correct.
Scheduling the pipeline as a Databricks Job enables automation and scalability, which are key features of a reusable pipeline.
- E. Incorrect.
Writing all pipeline logic in a single notebook reduces modularity and makes the pipeline harder to maintain and scale.