MLA-C01 Question 25
Single answerYou are building a machine learning pipeline on AWS to process a dataset that contains a large number of records with nested structures. The dataset needs to be stored in a format that supports efficient compression and schema evolution, while also enabling fast read performance for downstream processing in Amazon SageMaker. Which format should you choose?
- A
CSV
- B
JSON
- C
Apache Parquet
- D
Apache ORC
Show answer and explanation
Correct answer: C
Explanation
Apache Parquet is a columnar storage format that supports efficient compression, schema evolution, and fast data access for large-scale analytics and machine learning pipelines. These features make it well-suited for scenarios involving nested datasets and integration with AWS services such as Amazon SageMaker. Other formats like CSV and JSON lack the efficiency and scalability needed for this use case.
- A. Incorrect.
CSV is a simple, non-validated format that does not support efficient compression, schema evolution, or optimized read performance. It is not suitable for large-scale, nested datasets.
- B. Incorrect.
JSON supports nested structures, but it lacks efficient compression and optimized querying capabilities needed for fast read performance, making it less optimal for this use case.
- C. Correct.
Apache Parquet supports efficient compression, schema evolution, and optimized columnar storage, making it ideal for large-scale machine learning pipelines with nested data.
- D. Incorrect.
Apache ORC, while similar to Parquet, is less commonly used with Amazon SageMaker and the AWS ecosystem, and Parquet offers better compatibility with tools like AWS Glue and SageMaker.