DEA-C01 Question 98
Select 2You are designing a data pipeline on AWS to process daily log files uploaded to an S3 bucket. The pipeline should trigger a data processing workflow only after all log files for a day have been uploaded. The workflow should also run at a specific time each day. Which combination of services and configurations should you use to meet these requirements?
- A
Use Amazon S3 Event Notifications to trigger an AWS Lambda function for each log file uploaded and process them individually.
- B
Use AWS Step Functions with a 'Wait' state to implement a time-based delay and process logs after all files are uploaded.
- C
Use Amazon EventBridge to define a rule with a daily schedule to trigger an AWS Lambda function that checks if all log files are uploaded.
- D
Use an Amazon S3 Event Notification to write file upload events to an Amazon SQS queue and have an application poll the queue to check if all files are uploaded.
- E
Use AWS Glue Workflows to orchestrate the pipeline execution based on both time and file upload dependencies.
Show answer and explanation
Correct answers: C, E
Explanation
To meet the requirements, the pipeline must handle both time-based scheduling and file upload dependencies. Amazon EventBridge can be used to trigger a process daily to check if all files are uploaded, while AWS Glue Workflows provide a powerful orchestration framework for managing dependencies and scheduling within ETL workflows. Combining these services ensures that the pipeline runs correctly and efficiently.
- A. Incorrect.
This option processes each file individually as it is uploaded, which does not meet the requirement to wait until all files are uploaded for the day.
- B. Incorrect.
Step Functions can be used for orchestration, but managing the 'Wait' state for daily schedules and ensuring all files are uploaded is overly complex and not ideal for this use case.
- C. Correct.
Amazon EventBridge can be used to create a rule with a daily schedule to trigger a Lambda function that checks S3 for all required files and starts the pipeline if the condition is met. This approach satisfies both the time-based trigger and the dependency on file uploads.
- D. Incorrect.
Using S3 Event Notifications with SQS allows for tracking file upload events, but additional logic is required to ensure all files are uploaded and to handle time-based scheduling, making this approach incomplete.
- E. Correct.
AWS Glue Workflows are specifically designed to orchestrate ETL pipelines with dependencies. You can configure Glue Workflows to handle both time-based triggers and file upload dependencies, making it a suitable solution.