MLS-C01 Question 169
Single answerYou are building a machine learning pipeline to process a large dataset containing terabytes of semi-structured data. The pipeline requires distributed data processing, iterative machine learning model training, and tight integration with other AWS services. Which compute platform should you choose for this use case?
- A
Amazon EMR with Apache Spark
- B
AWS Lambda
- C
Amazon SageMaker Processing
- D
Amazon EC2
Show answer and explanation
Correct answer: A
Explanation
For a large-scale distributed data processing and machine learning pipeline, Amazon EMR with Apache Spark is the most appropriate choice. It is optimized for processing massive datasets, supports iterative workloads, and integrates easily with AWS services like S3 and SageMaker. Other options, such as AWS Lambda and SageMaker Processing, are not designed for distributed data processing or iterative workflows, while EC2 lacks built-in support for distributed frameworks.
- A. Correct.
Amazon EMR with Apache Spark is highly suitable for distributed data processing of large datasets and iterative workloads, such as machine learning model training. It also integrates well with AWS services.
- B. Incorrect.
AWS Lambda is not suitable for this use case because it is designed for short, stateless tasks and has limitations on execution duration and resources, making it unsuitable for large-scale distributed machine learning pipelines.
- C. Incorrect.
Amazon SageMaker Processing is great for preprocessing data, but it is not designed for large-scale distributed data processing or iterative machine learning model training.
- D. Incorrect.
Amazon EC2 provides flexible compute capacity but does not natively support distributed data processing frameworks like Apache Spark, making it less efficient for this use case.