MLA-C01 Question 24
Single answerYou are designing a machine learning pipeline to process a large dataset for training a model. The dataset contains millions of records with nested structures. The pipeline must efficiently handle schema evolution, support compression, and allow quick retrieval of specific columns for training. Which data format should you choose?
- A
CSV
- B
JSON
- C
Apache Parquet
- D
Apache Avro
Show answer and explanation
Correct answer: C
Explanation
For large datasets with nested structures, Apache Parquet is ideal as it is a columnar storage format that supports schema evolution and compression. These features allow for efficient storage and quick retrieval of specific columns, which are critical for machine learning pipelines. While formats like CSV, JSON, or Apache Avro have their use cases, they do not provide the same efficiency and flexibility as Apache Parquet for this scenario.
- A. Incorrect.
CSV is a non-validated format that does not support schema evolution or compression efficiently. It also requires scanning the entire file to retrieve specific columns.
- B. Incorrect.
JSON supports nested structures but is not optimized for columnar storage, making it inefficient for large-scale column retrieval. Additionally, JSON does not inherently support schema evolution.
- C. Correct.
Apache Parquet is a columnar storage format that supports schema evolution, efficient compression, and fast retrieval of specific columns, making it highly suitable for the described use case.
- D. Incorrect.
Apache Avro supports schema evolution and is efficient for row-based storage but is less optimized for columnar operations compared to Apache Parquet.