Databricks Machine Learning Professional Question 5
Select 2A data scientist is working on a machine learning project in Databricks and needs to manage multiple versions of datasets used for training models. The team wants to ensure reproducibility of results and track lineage of the datasets. Which of the following approaches should the data scientist prioritize?
- A
Use Delta Lake to version datasets and maintain a history of changes.
- B
Store datasets in external cloud storage without versioning for simplicity.
- C
Leverage Databricks Feature Store for tracking feature lineage and reuse.
- D
Manually log dataset metadata and versions in a spreadsheet for reference.
- E
Enable Auto Loader in Databricks to automatically track dataset changes.
Show answer and explanation
Correct answers: A, C
Explanation
To ensure reproducibility and track lineage of datasets in a machine learning workflow, leveraging Delta Lake for dataset versioning and Databricks Feature Store for feature lineage tracking are the most appropriate solutions. These tools are specifically designed to address challenges in data management for machine learning projects.
- A. Correct.
Delta Lake is a powerful tool for versioning datasets and maintaining a history of changes, which ensures reproducibility and dataset lineage tracking. This is a recommended approach.
- B. Incorrect.
Storing datasets in external cloud storage without versioning does not provide reproducibility or lineage tracking, making it unsuitable for this scenario.
- C. Correct.
Databricks Feature Store is designed to manage and track features and their lineage, which aligns with the goal of reproducibility and lineage tracking in machine learning projects.
- D. Incorrect.
Manually logging dataset metadata in a spreadsheet is error-prone, inefficient, and lacks automation, making it an impractical approach.
- E. Incorrect.
Auto Loader is useful for ingesting and processing streaming data but does not inherently provide dataset versioning or lineage tracking.