MLA-C01 Question 22
Select 2You are building a machine learning pipeline on AWS to process a large dataset. The dataset is semi-structured and frequently queried with complex analytical operations. You want to optimize both storage efficiency and query performance. Which data format(s) should you choose to store the dataset in Amazon S3?
- A
Apache Parquet
- B
JSON
- C
CSV
- D
Apache ORC
Show answer and explanation
Correct answers: A, D
Explanation
When dealing with semi-structured data and the need for optimized query performance and storage efficiency, columnar storage formats like Apache Parquet and Apache ORC are ideal. They allow for efficient compression, schema evolution, and fast analytical queries, making them well-suited for machine learning pipelines. JSON and CSV, while widely used, are not designed for such optimizations and may lead to higher costs and slower performance.
- A. Correct.
Apache Parquet is a columnar storage format optimized for analytical queries and storage efficiency, making it suitable for this scenario.
- B. Incorrect.
JSON is a semi-structured format, but it is not optimized for query performance or storage efficiency. It may result in higher storage costs and slower queries.
- C. Incorrect.
CSV is a simple format, but it lacks support for schema evolution and is not optimized for analytical queries or storage efficiency.
- D. Correct.
Apache ORC is a columnar storage format similar to Parquet, providing excellent query performance and storage efficiency for analytical workloads.