DEA-C01 Question 28
Single answerYou are designing a batch data ingestion pipeline for a financial application that processes transaction logs uploaded daily to an Amazon S3 bucket. The logs need to be ingested into an Amazon Redshift cluster for analytics. Your solution should optimize costs while ensuring the ingestion process runs automatically without manual intervention. Which solution best meets these requirements?
- A
Use AWS Glue to create a crawler that discovers the schema in the S3 bucket and triggers an ETL job to load the data into Amazon Redshift.
- B
Use Amazon Kinesis Data Firehose to stream data from the S3 bucket into Amazon Redshift.
- C
Schedule an AWS Lambda function using Amazon EventBridge to copy the data from S3 to Amazon Redshift using the COPY command.
- D
Use AWS DataSync to transfer data from S3 to Amazon Redshift on a daily schedule.
Show answer and explanation
Correct answer: A
Explanation
AWS Glue is the most suitable choice for batch data ingestion from Amazon S3 to Amazon Redshift in this scenario. It can automate schema discovery and manage ETL jobs, removing the need for manual intervention. The other options are either designed for different purposes (e.g., real-time streaming with Kinesis Firehose) or are not cost-efficient and scalable for this use case (e.g., Lambda or DataSync).
- A. Correct.
This is the correct option. AWS Glue can automate schema discovery and trigger an ETL job to transform and load the data into Amazon Redshift. It is cost-efficient and well-suited for batch ingestion from S3.
- B. Incorrect.
This is incorrect because Amazon Kinesis Data Firehose is designed for near real-time streaming data ingestion, not batch processing.
- C. Incorrect.
This is incorrect because AWS Lambda is not ideal for large data transfers due to its execution time and resource limits. Additionally, directly invoking the COPY command from Lambda for batch ingestion is not a cost-efficient or scalable approach.
- D. Incorrect.
This is incorrect because AWS DataSync is designed for large-scale data transfers between on-premises storage and AWS or between AWS storage services, and it is not optimized for loading data into Amazon Redshift.