DEA-C01 Question 26
Single answerYou are designing a batch data ingestion pipeline for a retail company that collects daily transaction data from multiple stores. The data needs to be ingested into Amazon S3 every night at 11 PM for downstream processing. Which approach would be the most appropriate to implement this requirement?
- A
Use AWS Lambda with an S3 event trigger to upload the data as soon as it arrives.
- B
Set up an Amazon EventBridge rule to trigger an AWS Glue ETL job at 11 PM to fetch data from stores and load it into Amazon S3.
- C
Use an Amazon S3 Transfer Acceleration endpoint to upload data from the stores directly to S3 as soon as it is generated.
- D
Configure an Amazon EC2 instance with a cron job that runs daily at 11 PM to collect transaction data and upload it to Amazon S3.
Show answer and explanation
Correct answer: B
Explanation
For scheduled batch data ingestion, Amazon EventBridge is the most suitable solution as it can trigger an AWS Glue ETL job at a specific time (11 PM) without requiring manual infrastructure management. This approach leverages fully managed services, aligning with best practices for scalability and cost-effectiveness.
- A. Incorrect.
AWS Lambda with an S3 event trigger is event-driven and not suitable for scheduled batch ingestion at a fixed time like 11 PM.
- B. Correct.
Amazon EventBridge allows you to define scheduled rules, making it a suitable choice for triggering an AWS Glue ETL job at a specific time like 11 PM for batch ingestion.
- C. Incorrect.
Amazon S3 Transfer Acceleration is used to speed up uploads to S3 but does not address the requirement of scheduling ingestion at a specific time.
- D. Incorrect.
Using an Amazon EC2 instance with a cron job is a valid approach, but it involves managing and maintaining an EC2 instance, which adds operational overhead compared to serverless and managed solutions like EventBridge and AWS Glue.