Databricks Machine Learning Professional Question 6
Select 2You are working on a machine learning project in Databricks, where your team needs to manage large amounts of labeled and unlabeled data stored across multiple cloud storage systems. The team wants to ensure that the data is correctly versioned, easily accessible for experiments, and that the lineage of the data transformations is maintained. Which of the following strategies would best meet these requirements?
- A
Use Databricks Delta Lake to store and version the data, enabling time travel for accessing previous versions of the data.
- B
Manually copy and store different versions of the data files in separate cloud storage folders to track changes.
- C
Leverage Databricks feature store for storing and versioning datasets used in machine learning experiments.
- D
Use a centralized metadata store to track data lineage and transformations across the machine learning pipeline.
- E
Rely solely on Databricks MLflow to manage data transformations and versioning.
Show answer and explanation
Correct answers: A, D
Explanation
To effectively manage large amounts of data in Databricks for machine learning, Delta Lake provides a reliable solution for data versioning and time travel, ensuring previous versions of the data can be accessed. A centralized metadata store complements this by tracking lineage and transformations, which is critical for maintaining data integrity and auditability throughout the ML pipeline. Together, these strategies address the requirements of versioning, accessibility, and lineage management.
- A. Correct.
Databricks Delta Lake provides built-in support for data versioning and time travel, enabling easy access to previous states of the data. This satisfies the requirement for versioning and simplifies access for experiments.
- B. Incorrect.
Manually copying and storing versions of data in separate folders is error-prone, labor-intensive, and does not provide an efficient mechanism for managing data lineage or versioning.
- C. Incorrect.
The Databricks feature store is designed for managing machine learning features, not for storing and versioning datasets. It is not suitable for this use case.
- D. Correct.
A centralized metadata store is essential for tracking data lineage and transformations, ensuring the team can understand and audit how the data evolves through the pipeline.
- E. Incorrect.
MLflow is primarily used for experiment tracking, model management, and deployment. It does not natively support data versioning or lineage for datasets.