Databricks Machine Learning Professional Question 143
Single answerYou are tasked with training a machine learning model on a dataset stored in a data lake using Databricks. During initial experimentation, you notice high latency when reading data for your training jobs. Upon investigation, you discover that the dataset is stored in a file format that is not optimized for analytics workloads. Which of the following file formats would be less performant for this use case, but may still be appropriate for other use cases such as raw data storage or archival purposes?
- A
CSV
- B
Parquet
- C
Delta Lake
- D
ORC
Show answer and explanation
Correct answer: A
Explanation
CSV is less performant for machine learning workloads compared to columnar storage formats like Parquet, Delta Lake, and ORC. While CSV is not optimized for analytics, it can still be a good choice for simple data storage or scenarios where compatibility is a priority. Machine learning workloads typically require formats that improve I/O performance and support advanced features like schema evolution and compression.
- A. Correct.
CSV is a plain text format that is not optimized for analytics workloads. It lacks columnar storage and indexing, making it slower for reading and querying large datasets. However, it is simple, widely supported, and suitable for raw data storage or archival.
- B. Incorrect.
Parquet is a columnar storage format designed for high-performance analytics. It is optimized for reading and querying large datasets, making it a suitable choice for machine learning workloads.
- C. Incorrect.
Delta Lake is an optimized storage layer built on top of Parquet, offering additional features like ACID transactions and time travel. It is highly performant and well-suited for machine learning workloads.
- D. Incorrect.
ORC is another columnar storage format optimized for big data analytics workloads. It provides efficient data compression and performance, making it appropriate for machine learning tasks.