DEA-C01 Question 88
Select 2You are designing an ETL pipeline for processing log files from an S3 bucket. The pipeline needs to transform the data and store it in an Amazon Redshift table for analytics. The log files arrive in the S3 bucket at irregular intervals. The solution should be cost-effective and fully managed. Which combination of AWS services would best meet these requirements?
- A
Use AWS Glue to crawl the data in the S3 bucket, transform it, and load it into Amazon Redshift.
- B
Use Amazon Kinesis Data Firehose to stream data directly from the S3 bucket to Amazon Redshift.
- C
Use an AWS Lambda function to trigger on new S3 objects, transform the data, and insert it into Amazon Redshift.
- D
Use AWS Glue workflows to orchestrate the ETL process and AWS Glue jobs to perform the transformations.
- E
Use Amazon EMR to process the data in S3 and save the output to an Amazon Redshift table.
Show answer and explanation
Correct answers: A, D
Explanation
The best solution for this scenario is to use AWS Glue for both the ETL transformations and orchestration. AWS Glue is designed to handle ETL processes efficiently, integrates well with S3 and Amazon Redshift, and is fully managed, making it cost-effective. Other options like Amazon Kinesis Data Firehose and AWS Lambda are not suitable for batch processing of log files, and Amazon EMR is unnecessarily complex and costly for this use case.
- A. Correct.
Correct: AWS Glue can crawl the schema of the data in the S3 bucket, perform the necessary transformations, and load the data into Amazon Redshift. This approach is fully managed and cost-effective.
- B. Incorrect.
Incorrect: Amazon Kinesis Data Firehose is suitable for streaming data, but the log files are already stored in S3 and arrive at irregular intervals. This makes Kinesis Data Firehose less suitable for this use case.
- C. Incorrect.
Incorrect: While AWS Lambda can trigger on new S3 objects and perform lightweight transformations, it is not ideal for complex ETL processes, and it cannot directly load large datasets into Amazon Redshift.
- D. Correct.
Correct: AWS Glue workflows can be used to orchestrate the ETL process, and Glue jobs can handle the data transformations. AWS Glue is a fully managed, serverless solution that fits the requirements.
- E. Incorrect.
Incorrect: Amazon EMR is a powerful tool for complex big data processing but is not as cost-effective or fully managed as AWS Glue for this scenario.