Databricks Machine Learning Associate Question 564
Single answerYou are tasked with developing a training pipeline for a machine learning model on Databricks. The pipeline involves data preprocessing, model training, and evaluation. Which step is MOST critical to ensure the pipeline is reusable and parameterized for different datasets and hyperparameter configurations?
- A
Using Databricks widgets to collect user inputs during pipeline execution
- B
Hard-coding dataset paths and hyperparameters directly into the pipeline script
- C
Writing the pipeline as a series of modular, reusable functions
- D
Saving the trained model within the pipeline without logging any metadata
Show answer and explanation
Correct answer: C
Explanation
Developing a reusable and parameterized training pipeline is crucial for production-level machine learning workflows. Writing the pipeline as modular, reusable functions ensures flexibility and adaptability for different use cases, such as handling multiple datasets or experimenting with various hyperparameters. This approach aligns with best practices for building robust machine learning pipelines in Databricks.
- A. Incorrect.
Using Databricks widgets to collect user inputs can be useful for interactivity but does not inherently make the pipeline reusable or parameterized. It is more appropriate for ad-hoc experimentation than production pipelines.
- B. Incorrect.
Hard-coding dataset paths and hyperparameters reduces flexibility and reusability, making the pipeline unsuitable for different datasets or configurations.
- C. Correct.
Writing the pipeline as a series of modular, reusable functions ensures that the pipeline can easily adapt to different datasets and configurations by simply passing different parameters. This is central to creating a reusable and robust training pipeline.
- D. Incorrect.
Saving the trained model without logging metadata can lead to challenges in model management and reproducibility. Logging metadata is essential for traceability and reproducibility of the pipeline.