DEA-C01 Question 100
Select 2An e-commerce company wants to process daily transaction logs stored in Amazon S3 and load the aggregated data into Amazon Redshift for analytics. The processing should start every day at 1 AM UTC and should wait for all transaction logs to be uploaded before beginning. Which combination of AWS services and configurations should be used to implement this pipeline?
- A
Use AWS Glue workflows to orchestrate the ETL process, configure a trigger to start the workflow at 1 AM UTC, and define a condition to check for the presence of all required files in Amazon S3.
- B
Use Amazon EventBridge to schedule a rule that triggers an AWS Lambda function at 1 AM UTC, which verifies the transaction files in Amazon S3 and invokes an AWS Batch job to process the data.
- C
Use AWS Step Functions to create a state machine, configure a scheduled start at 1 AM UTC, and add a custom Lambda function to check for file availability before starting the ETL process.
- D
Configure Amazon S3 Event Notifications to trigger the pipeline as soon as each transaction log file is uploaded, and use AWS Glue jobs to process and aggregate the data.
- E
Use Amazon Data Pipeline to schedule a daily activity at 1 AM UTC, configure a precondition to check for the presence of all required transaction log files in Amazon S3, and use an activity to load the data into Amazon Redshift.
Show answer and explanation
Correct answers: A, E
Explanation
To implement a data pipeline that is both scheduled and dependent on the presence of all required files in Amazon S3, AWS Glue workflows and Amazon Data Pipeline are the most suitable options. AWS Glue workflows provide a serverless way to orchestrate ETL processes with triggers and conditions. Amazon Data Pipeline offers similar capabilities for dependency checking and scheduling but requires more configuration. Both approaches satisfy the requirements of starting the pipeline at 1 AM UTC and ensuring all files are available before processing.
- A. Correct.
Correct: AWS Glue workflows allow orchestration of ETL processes with triggers and conditional checks, making it suitable for scheduled pipelines dependent on file availability.
- B. Incorrect.
Partially correct but less optimal: While Amazon EventBridge and Lambda can handle scheduling and file verification, incorporating AWS Batch adds unnecessary complexity for this use case.
- C. Incorrect.
Incorrect: AWS Step Functions can manage workflows and scheduling, but it is less suited for complex ETL processes compared to AWS Glue or Amazon Data Pipeline.
- D. Incorrect.
Incorrect: Using S3 Event Notifications triggers the pipeline for every file upload, which does not meet the requirement for starting the process at a specific time and ensuring all files are present.
- E. Correct.
Correct: Amazon Data Pipeline supports scheduled workflows with preconditions, making it a good choice for handling dependencies like file availability in Amazon S3.