MLS-C01 Question 61
Single answerYou are working for a retail company that collects extensive customer transaction data daily. The dataset size exceeds several terabytes, and you need to preprocess this data to remove null values, aggregate sales per region, and prepare it for training a machine learning model. The company uses Amazon EMR for big data processing. Which of the following approaches would be the most efficient for preprocessing the data?
- A
Use Apache Spark on Amazon EMR to load the dataset, perform transformations to clean and aggregate the data, and output the results to Amazon S3.
- B
Use Amazon SageMaker Data Wrangler to load the dataset directly from Amazon S3 and perform the required preprocessing tasks.
- C
Use Amazon Athena to query the dataset in Amazon S3 and write the results back into a new S3 bucket after the preprocessing steps.
- D
Use AWS Glue to preprocess the data by creating an ETL job and directly integrate the output with the machine learning model.
Show answer and explanation
Correct answer: A
Explanation
Apache Spark on Amazon EMR is designed for distributed big data processing and is well-suited for preprocessing tasks on large-scale datasets. It provides the performance and scalability required to handle terabyte-scale data while offering flexibility for advanced transformations. The other options, while useful for certain use cases, are either not optimized for this scale or lack the required capabilities for efficient preprocessing of very large datasets.
- A. Correct.
This is the correct answer. Apache Spark on Amazon EMR is optimized for large-scale distributed data processing, making it suitable for handling terabyte-scale datasets efficiently. It allows for complex transformations and aggregation, and the processed data can be saved back to Amazon S3.
- B. Incorrect.
Incorrect. While Amazon SageMaker Data Wrangler is a powerful tool for data preparation, it is not designed to handle datasets of this scale efficiently. It is better suited for smaller datasets or those that are already partially preprocessed.
- C. Incorrect.
Incorrect. Amazon Athena is a serverless query service suitable for ad-hoc querying but is not ideal for complex data transformations or large-scale preprocessing tasks. It is also less efficient for continuous preprocessing workflows.
- D. Incorrect.
Incorrect. AWS Glue is a managed ETL service, but it is not specifically optimized for high-performance distributed data transformations like Apache Spark on Amazon EMR. It may introduce higher latency and lacks the flexibility of Spark for this scale of data processing.