MLA-C01 Question 23
Select 3A data engineering team at a large organization is preparing datasets to train a machine learning model on AWS SageMaker. The team needs to store the training data in a format that ensures efficient performance during distributed training jobs and minimizes storage costs. Which of the following data formats are most suitable for this use case?
- A
Apache Parquet
- B
CSV
- C
JSON
- D
Apache ORC
- E
RecordIO
Show answer and explanation
Correct answers: A, D, E
Explanation
For distributed machine learning training jobs, efficiency in storage and processing is critical. Columnar formats like Apache Parquet and Apache ORC are optimized for efficient storage and distributed processing, making them suitable for this use case. RecordIO is specifically designed for machine learning workloads and integrates seamlessly with AWS SageMaker, making it an excellent choice for training data. CSV and JSON, while common, are less efficient for distributed training due to their row-based structure and lack of optimization for distributed workloads.
- A. Correct.
Apache Parquet is a columnar storage format that is highly efficient for both storage and distributed processing, making it suitable for training machine learning models.
- B. Incorrect.
CSV is a row-based format, which is less efficient for distributed processing and does not support advanced compression or schema evolution, making it less ideal for this scenario.
- C. Incorrect.
JSON is a semi-structured, text-based format that is not optimized for large-scale distributed processing or storage efficiency, making it unsuitable for this use case.
- D. Correct.
Apache ORC is a columnar storage format similar to Parquet and is optimized for efficient storage and distributed processing, making it suitable for training machine learning models.
- E. Correct.
RecordIO is a binary record-based format that is highly optimized for training machine learning models on AWS SageMaker, especially for distributed training.