MLA-C01 Question 21
Select 3You are building a machine learning pipeline on AWS to process large-scale datasets and prepare them for training. The datasets are stored in Amazon S3, and you want to choose a storage format that optimizes for both read performance and efficient storage. Additionally, you need a format that supports schema evolution and is well-suited for distributed processing frameworks such as Apache Spark. Which of the following options meet these requirements?
- A
Apache Parquet
- B
JSON
- C
Apache ORC
- D
CSV
- E
Apache Avro
Show answer and explanation
Correct answers: A, C, E
Explanation
Columnar formats like Apache Parquet and Apache ORC are optimized for high read performance and storage efficiency, making them ideal for large-scale machine learning pipelines. Apache Avro, while row-based, supports schema evolution and works well with distributed processing frameworks, making it another viable choice. JSON and CSV, being text-based formats, lack the necessary performance optimizations and features required for ML pipelines at scale.
- A. Correct.
Apache Parquet is a columnar storage format that is optimized for both query performance and storage efficiency. It also supports schema evolution and works well with distributed processing frameworks, making it a suitable choice.
- B. Incorrect.
JSON is a text-based format that is not optimized for read performance or storage efficiency. While it is widely used for data interchange, it lacks the performance benefits of formats like Parquet or ORC.
- C. Correct.
Apache ORC is a columnar storage format similar to Parquet, offering high read performance and efficient storage. It supports schema evolution and is compatible with distributed processing frameworks, making it a suitable choice.
- D. Incorrect.
CSV is a simple text-based format that lacks advanced features like schema evolution, storage efficiency, and performance optimization. It is not ideal for large-scale ML pipelines.
- E. Correct.
Apache Avro is a row-based storage format that supports schema evolution and is designed for efficient data serialization. While it is not columnar, it is still a good choice for distributed processing frameworks and supports large-scale ML pipelines.