Databricks Machine Learning Professional Question 140
Select 2A company is using Databricks to train machine learning models on large datasets stored in Delta Lake. However, the team identified that some of the intermediate data generated during the preprocessing phase (e.g., temporary feature extraction results) does not require high-performance storage and can be moved to a less performant storage to optimize costs. Which storage option(s) would be suitable for this use case?
- A
Object storage solutions like AWS S3 or Azure Blob Storage
- B
Delta Lake with optimized Delta cache enabled
- C
HDFS (Hadoop Distributed File System) on high-performance clusters
- D
Databricks Filesystem (DBFS) mounted to cloud object storage
- E
External databases like MySQL or PostgreSQL
Show answer and explanation
Correct answers: A, D
Explanation
For intermediate or temporary data that does not require high-performance storage, cost-effective solutions like object storage (e.g., AWS S3 or Azure Blob Storage) or DBFS mounted to cloud object storage are ideal. These options are scalable, cost-efficient, and well-integrated with Databricks workflows, unlike high-performance options like Delta Lake or HDFS, which may unnecessarily increase costs for temporary data storage.
- A. Correct.
Object storage solutions like AWS S3 or Azure Blob Storage are cost-effective and suitable for storing non-critical or intermediate data, even though their performance is lower than high-speed storage options.
- B. Incorrect.
Delta Lake with optimized Delta cache enabled is designed for high-performance querying and storage, making it a suboptimal choice for non-critical intermediate data.
- C. Incorrect.
HDFS on high-performance clusters provides excellent performance for data processing but is expensive and not cost-efficient for temporary or intermediate data storage.
- D. Correct.
Databricks Filesystem (DBFS) mounted to cloud object storage leverages the underlying cost-effectiveness of object storage while being accessible through Databricks APIs, making it a good choice for intermediate data storage.
- E. Incorrect.
External databases like MySQL or PostgreSQL are not suitable for storing intermediate data from preprocessing due to their high operational complexity and lack of scalability for large, unstructured or semi-structured datasets.