MLS-C01 Question 62
Single answerYou are working on a machine learning project that requires preprocessing a large dataset stored in Amazon S3. The dataset needs to be cleaned, aggregated, and transformed before training a model. Due to its size, you want to leverage a distributed data processing framework to handle the operations efficiently. Which solution is the most appropriate for this scenario?
- A
Use AWS Glue with an Apache Spark environment to preprocess the data in a distributed manner.
- B
Download the dataset to an Amazon EC2 instance and preprocess it locally using Python scripts.
- C
Use Amazon Redshift to preprocess the data by writing SQL queries.
- D
Use Amazon EMR with Apache Hive to preprocess the data and store the results back in Amazon S3.
Show answer and explanation
Correct answer: A
Explanation
To preprocess large datasets stored in Amazon S3 for machine learning, a distributed data processing framework is necessary. AWS Glue, which is based on Apache Spark, provides a scalable, serverless solution for distributed ETL tasks. It is optimized for working with S3 and is more suitable than alternatives like local processing or Amazon Redshift for this scenario.
- A. Correct.
This is the correct answer. AWS Glue provides a serverless Apache Spark environment that is well-suited for distributed data preprocessing tasks. It is scalable and integrates seamlessly with S3 for handling large datasets.
- B. Incorrect.
Downloading the dataset to a single EC2 instance and processing it locally is not scalable for large datasets, as it lacks distributed computing capabilities.
- C. Incorrect.
While Amazon Redshift can handle large-scale data warehousing and allow SQL-based operations, it is not designed specifically for distributed preprocessing of datasets in the same way as Spark or Hive.
- D. Incorrect.
Amazon EMR with Apache Hive can preprocess data, but it is not as efficient or flexible as using AWS Glue with Spark for this use case. AWS Glue is purpose-built for distributed ETL tasks and integrates better with S3.