SAA-C03 Question 253
Single answerA company needs to process a large number of data files for a machine learning training pipeline. The processing requires distributed data transformation and cleaning tasks that can be executed in parallel. The company wants to minimize operational overhead while ensuring scalability and cost-efficiency. Which AWS service is the most appropriate for this use case?
- A
AWS Batch
- B
Amazon EMR
- C
AWS Fargate
- D
AWS Lambda
Show answer and explanation
Correct answer: B
Explanation
Amazon EMR is the most appropriate service for this scenario because it is specifically designed for distributed big data processing tasks, such as data transformation and cleaning, using frameworks like Apache Spark and Hadoop. It provides scalability and cost-efficiency while minimizing operational overhead, which aligns perfectly with the company's requirements.
- A. Incorrect.
AWS Batch is suitable for batch computing jobs, but it does not provide the same level of native tooling for distributed big data processing and is less specialized for data transformation workloads compared to Amazon EMR.
- B. Correct.
Amazon EMR (Elastic MapReduce) is designed for distributed big data processing using frameworks like Apache Spark and Hadoop. It is ideal for processing large datasets in parallel, making it the best choice for this machine learning data pipeline.
- C. Incorrect.
AWS Fargate is a serverless compute engine for containers. While it simplifies container management, it is not designed specifically for distributed data processing workloads.
- D. Incorrect.
AWS Lambda is a serverless compute service for running short-duration, event-driven tasks. It is not optimized for large-scale, distributed data transformation and is limited by execution time and memory constraints.