Databricks Machine Learning Associate Question 563
Select 4You are tasked with building a scalable machine learning training pipeline in Databricks. The pipeline should preprocess data, train a model, and log metrics for reproducibility. Which of the following steps are necessary to develop this pipeline using Databricks best practices?
- A
Use MLflow to log metrics, parameters, and the trained model.
- B
Store intermediate data in a local file system for better performance.
- C
Leverage Databricks Jobs to orchestrate the training pipeline.
- D
Perform feature engineering and model training within a single notebook without modularization.
- E
Use Delta Lake for version-controlled data storage during preprocessing.
- F
Automate hyperparameter tuning with MLflow or Databricks-managed tools.
Show answer and explanation
Correct answers: A, C, E, F
Explanation
To develop a scalable training pipeline in Databricks, you should follow best practices such as using MLflow for experiment tracking, leveraging Delta Lake for version-controlled data storage, and automating processes like hyperparameter tuning. Databricks Jobs provide an effective mechanism to orchestrate the entire pipeline. Avoid practices like storing intermediate data locally or combining all tasks within a single notebook, as these hinder scalability and maintainability.
- A. Correct.
Correct: MLflow is an integral tool in Databricks for tracking experiments, logging metrics, and managing trained models.
- B. Incorrect.
Incorrect: Storing intermediate data in the local file system is not recommended in Databricks because it does not support distributed processing or scalability.
- C. Correct.
Correct: Databricks Jobs are used to automate and orchestrate machine learning pipelines effectively.
- D. Incorrect.
Incorrect: Combining feature engineering and model training in a single notebook without modularization is against best practices as it reduces code reusability and maintainability.
- E. Correct.
Correct: Delta Lake is a Databricks-native tool that provides versioning and ACID transactions, making it ideal for data preprocessing in pipelines.
- F. Correct.
Correct: Automating hyperparameter tuning improves the efficiency of the training pipeline and is supported by Databricks through tools like Hyperopt and MLflow integration.