MLS-C01 Question 171
Single answerYou are a data scientist working at a company that processes large-scale clickstream data. The team wants to preprocess and analyze this data to build a recommendation system. The dataset is stored in Amazon S3, consists of hundreds of terabytes, and requires distributed data processing. Which compute platform would be the most appropriate to use for this workload?
- A
Amazon EMR with Apache Spark
- B
AWS Lambda
- C
Amazon SageMaker Processing
- D
Amazon EC2 with Python scripts
Show answer and explanation
Correct answer: A
Explanation
When working with large-scale datasets (such as hundreds of terabytes) stored in Amazon S3, distributed data processing is essential to handle the workload efficiently. Apache Spark running on Amazon EMR is specifically designed for such use cases, providing a scalable and distributed compute platform. Other options either lack the required scalability or are not optimized for large-scale distributed data processing.
- A. Correct.
Amazon EMR with Apache Spark is well-suited for distributed data processing of large-scale datasets. Apache Spark is specifically designed to handle big data workloads in a distributed and efficient manner. Since the dataset consists of hundreds of terabytes, this is the most appropriate choice.
- B. Incorrect.
AWS Lambda is not suitable for processing massive datasets because it has strict resource and execution time limits. While it can handle simple tasks, it is not optimized for large-scale distributed data processing.
- C. Incorrect.
Amazon SageMaker Processing is designed for batch processing and preprocessing of data for machine learning, but it is not optimized for handling hundreds of terabytes of data in a distributed manner like Apache Spark.
- D. Incorrect.
Amazon EC2 with Python scripts could be used for smaller datasets or simple tasks, but it is not designed to efficiently process hundreds of terabytes of data in a distributed manner. This would require significant manual effort to set up and maintain.