MLS-C01 Question 4
Single answerA data engineering team is building a pipeline to preprocess and transform a large dataset for a machine learning model. The dataset is stored in Amazon S3, and the preprocessing involves operations such as filtering, aggregation, and normalization. The team wants to ensure the pipeline can scale, handle large volumes of data efficiently, and integrate with other AWS services to store the final processed data in Amazon Redshift for analytics. Which AWS service should the team use to efficiently perform these preprocessing tasks?
- A
AWS Glue
- B
Amazon SageMaker Processing
- C
AWS Lambda
- D
Amazon EMR
Show answer and explanation
Correct answer: A
Explanation
AWS Glue is the best choice for this scenario because it is a fully managed ETL service that is optimized for preprocessing large datasets stored in Amazon S3. It provides scalability, integrates with Amazon Redshift, and simplifies the data transformation process. While other options like Amazon EMR and SageMaker Processing can perform similar tasks, AWS Glue offers the most efficient and seamless solution for this use case.
- A. Correct.
AWS Glue is a fully managed ETL (Extract, Transform, Load) service that is designed for preprocessing large datasets. It integrates seamlessly with Amazon S3 and Amazon Redshift, making it the most suitable choice for this scenario.
- B. Incorrect.
Amazon SageMaker Processing is designed for preparing and processing data for machine learning workflows, but it is not a specialized ETL tool. It is more suited for ML-specific preprocessing rather than large-scale data engineering tasks.
- C. Incorrect.
AWS Lambda is a serverless compute service that can process data, but it is not well-suited for handling large datasets due to its memory and processing time limitations. It is better for event-driven tasks and lightweight operations.
- D. Incorrect.
Amazon EMR is a good choice for big data processing, but it requires more setup and management compared to AWS Glue. While it can perform the tasks described, AWS Glue provides a simpler and more integrated solution for this specific use case.