DEA-C01 Question 51
Select 2A retail company wants to build an ETL pipeline to process daily transaction logs stored in Amazon S3. The pipeline should filter out invalid transactions, enrich the data with customer details from Amazon DynamoDB, and store the transformed data into Amazon Redshift for analytics. Which combination of AWS services and features should you use to meet these requirements?
- A
AWS Glue for data extraction, transformation, and loading
- B
Amazon EMR with Spark for data transformation and enrichment
- C
AWS Glue DataBrew for enriching data with DynamoDB
- D
Amazon Redshift COPY command for loading transformed data
- E
AWS Lambda for filtering invalid transactions
Show answer and explanation
Correct answers: A, D
Explanation
To build a robust ETL pipeline for this use case, AWS Glue provides a serverless and scalable solution for extracting data from S3, transforming it (including enrichment from DynamoDB), and preparing it for loading. The Amazon Redshift COPY command is the optimal method for ingesting the transformed data into the data warehouse. Other options like EMR or Lambda are less suitable for these specific requirements due to complexity or limitations in scalability.
- A. Correct.
Correct: AWS Glue is a serverless ETL service that can extract, transform, and load data effectively from S3 to Amazon Redshift, while also integrating with external sources like DynamoDB for data enrichment.
- B. Incorrect.
Incorrect: While Amazon EMR can handle data transformation and enrichment, it is not the most serverless or cost-effective option compared to AWS Glue for the described use case. EMR is better suited for large-scale, custom distributed processing.
- C. Incorrect.
Incorrect: AWS Glue DataBrew is used for data preparation and cleaning in a visual interface, but it is not designed for dynamic enrichment with external databases like DynamoDB.
- D. Correct.
Correct: Amazon Redshift's COPY command is highly optimized for loading bulk data from Amazon S3 into Redshift tables, making it the appropriate choice for loading the transformed data.
- E. Incorrect.
Incorrect: AWS Lambda can be used for lightweight processing tasks, but it is not designed for complex ETL pipelines involving enrichment and loading into a data warehouse like Redshift.