DEA-C01 Question 76
Select 2You are a data engineer tasked with building an end-to-end data pipeline that ingests data from an S3 bucket, transforms it using AWS Glue, and loads it into an Amazon Redshift data warehouse. You also need to schedule this pipeline to run daily at midnight. Which combination of services and features should you use to orchestrate this workflow effectively?
- A
AWS Step Functions to define and manage the workflow
- B
Amazon EventBridge to trigger the pipeline execution at midnight
- C
AWS Glue Workflows to orchestrate the data transformation and loading steps
- D
AWS Lambda to schedule the pipeline execution
- E
Amazon S3 Event Notifications to trigger the pipeline when new data is added
Show answer and explanation
Correct answers: A, B
Explanation
To orchestrate an end-to-end data pipeline with time-based scheduling, you need a service for workflow orchestration (AWS Step Functions) and a service for scheduling (Amazon EventBridge). AWS Glue Workflows, while useful for orchestrating Glue-specific tasks, cannot manage an entire pipeline involving multiple services. S3 Event Notifications and AWS Lambda are not suitable for this scenario as they either cannot handle time-based triggers or are not designed for complex orchestration.
- A. Correct.
AWS Step Functions is a serverless orchestration service that helps define and manage complex workflows, making it suitable for orchestrating an end-to-end pipeline involving multiple services like AWS Glue and Redshift.
- B. Correct.
Amazon EventBridge is a serverless event bus that can be used to schedule the pipeline to run daily at midnight, fulfilling the scheduling requirement.
- C. Incorrect.
AWS Glue Workflows are useful for orchestrating steps within AWS Glue itself, but they are not equipped to orchestrate an end-to-end pipeline involving multiple services like S3 and Redshift.
- D. Incorrect.
AWS Lambda can be used to execute code in response to events, but it is not an ideal tool for scheduling or orchestrating complex workflows compared to EventBridge and Step Functions.
- E. Incorrect.
Amazon S3 Event Notifications can trigger workflows when new data is added to S3, but the question specifies a time-based trigger (midnight), which S3 Event Notifications cannot handle.