Databricks Machine Learning Professional Question 141
Single answerA data science team is working with a dataset that is accessed infrequently during model training but needs to be retained for future audits and compliance purposes. They are currently using a high-performance storage solution that incurs high costs, but they want to optimize costs by transferring the dataset to a less performant storage option. Which storage solution is the most appropriate for this use case?
- A
Databricks Delta Lake on high-performance SSD storage
- B
AWS S3 Standard storage class
- C
AWS S3 Glacier storage class
- D
Databricks DBFS (Databricks File System) on high-performance cluster storage
Show answer and explanation
Correct answer: C
Explanation
The dataset is accessed infrequently and needs to be retained for long-term compliance. AWS S3 Glacier is specifically designed for such use cases, offering low-cost storage with higher retrieval latency, which is acceptable for audit and compliance scenarios. High-performance storage options like Delta Lake or DBFS are unnecessary and cost-inefficient for this use case.
- A. Incorrect.
Databricks Delta Lake on high-performance SSD storage is designed for fast reads and writes, making it suitable for active workloads like frequent access during training or real-time analytics. This is not cost-effective for infrequent access scenarios.
- B. Incorrect.
AWS S3 Standard storage class is designed for frequent access patterns and offers lower latency but higher costs compared to archive storage solutions. It is not ideal for infrequently accessed datasets.
- C. Correct.
AWS S3 Glacier storage class is a cost-effective solution for infrequently accessed datasets that need to be retained for long-term purposes, such as audits and compliance. However, it has higher retrieval latency, which aligns well with the stated requirements.
- D. Incorrect.
Databricks DBFS on high-performance cluster storage is optimized for active workloads and fast access speeds. It is not a cost-efficient choice for storing infrequently accessed data.