Databricks Machine Learning Professional Question 142
Single answerA data science team is working on a machine learning project using Databricks. They are currently storing their feature data in a Delta table for efficient access and updates. However, they need to archive older, less frequently accessed data into a less performant, but cost-effective storage solution. Which of the following storage options would be the most appropriate for this use case?
- A
Amazon S3 with Parquet format
- B
Delta Lake with time travel enabled
- C
Databricks File System (DBFS) with JSON format
- D
Azure Blob Storage with CSV format
Show answer and explanation
Correct answer: A
Explanation
The most suitable solution for archiving older, less frequently accessed data is Amazon S3 with Parquet format. This option balances cost-effectiveness and acceptable performance for archival use cases. Parquet's columnar storage format is efficient for querying archived data when needed, while S3 offers scalable and affordable storage. Other options, like Delta Lake, are more performant but unnecessary for this use case, and formats like JSON or CSV are less efficient for large datasets.
- A. Correct.
Amazon S3 with Parquet format is a cost-effective and less performant storage solution compared to Delta Lake. Parquet is a columnar file format that optimizes storage and query performance for archived data, making it suitable for this use case.
- B. Incorrect.
Delta Lake with time travel enabled is designed for high-performance data operations and version control, which is unnecessary for archiving older data. It is not cost-effective for less frequently accessed data.
- C. Incorrect.
Databricks File System (DBFS) with JSON format is not ideal for archiving data due to the inefficiency of JSON as a storage format for large datasets. It provides neither cost optimization nor sufficient performance for this scenario.
- D. Incorrect.
Azure Blob Storage with CSV format is a less performant storage option but lacks the efficiency of columnar formats like Parquet. CSV files can consume more storage and lead to higher costs for large datasets.