DEA-C01 Question 54
Select 2Your company has a large dataset stored in Amazon S3, and you need to create an ETL pipeline to process this data daily. The business requirements specify that the pipeline should: 1) extract data from S3, 2) transform it by filtering records and aggregating data, 3) load the processed data into Amazon Redshift for analytics. The pipeline should be cost-effective and scalable. Which combination of AWS services should you use to meet these requirements?
- A
AWS Glue for the ETL process and Amazon Redshift for the analytics layer
- B
Amazon EMR for the ETL process and Amazon RDS for the analytics layer
- C
AWS Glue for data extraction and transformation, and Amazon Redshift for data loading and analytics
- D
Amazon Data Pipeline for orchestrating tasks and Amazon DynamoDB for storing transformed data
- E
AWS Lambda for data transformation and Amazon Redshift for storing processed data
Show answer and explanation
Correct answers: A, C
Explanation
The combination of AWS Glue and Amazon Redshift is the most appropriate choice for this scenario. AWS Glue provides a cost-effective, serverless solution for extracting and transforming data, while Amazon Redshift is optimized for large-scale analytics and can efficiently handle the processed data. This setup meets the business requirements of scalability and cost-effectiveness.
- A. Correct.
Correct: AWS Glue is a serverless ETL service that can extract data from Amazon S3, transform it, and load it into Amazon Redshift. It is cost-effective and scalable for this use case.
- B. Incorrect.
Incorrect: Amazon EMR can handle ETL processes, but Amazon RDS is not optimized for large-scale analytics compared to Amazon Redshift, which is purpose-built for data warehousing.
- C. Correct.
Correct: AWS Glue can handle data extraction and transformation, while Amazon Redshift is designed for analytics and can efficiently handle the processed data.
- D. Incorrect.
Incorrect: Amazon Data Pipeline is used for data workflow orchestration but is not a direct ETL tool. Additionally, Amazon DynamoDB is not suitable for analytics compared to Amazon Redshift.
- E. Incorrect.
Incorrect: AWS Lambda is not the best choice for a complex ETL pipeline involving large-scale transformations. While Amazon Redshift is appropriate for analytics, Lambda's limitations in runtime and memory make it unsuitable for this scenario.