DEA-C01 Question 96
Single answerYou are building a data pipeline to process log files from an S3 bucket and store the results in Amazon Redshift. The pipeline is scheduled to run at midnight every day. The logs must first be pre-processed using AWS Glue, and the output from Glue should be loaded into Redshift using an AWS Step Functions workflow. How can you ensure that the pipeline runs daily at midnight and that the Glue job completes successfully before the Redshift loading step begins?
- A
Use Amazon EventBridge to trigger the Step Functions workflow at midnight and configure the workflow to orchestrate the Glue job and Redshift loading step.
- B
Configure an Amazon S3 event notification to trigger the Glue job and then use AWS Lambda to load the output from Glue into Redshift.
- C
Set up an AWS Glue Workflow to schedule the Glue job and use AWS Batch to load the results into Redshift.
- D
Use AWS Data Pipeline to orchestrate the Glue job and Redshift loading step, and configure the pipeline's schedule to run at midnight.
Show answer and explanation
Correct answer: A
Explanation
Amazon EventBridge is used to create a scheduled rule to trigger workflows or jobs at specific times, such as midnight in this scenario. AWS Step Functions is ideal for orchestrating complex workflows with dependencies, ensuring that the Glue job is successfully completed before loading data into Redshift. This combination of services is a modern and serverless approach for building robust data pipelines.
- A. Correct.
This is the correct answer. Amazon EventBridge can trigger the Step Functions workflow on a daily schedule (e.g., midnight). Step Functions can then orchestrate the Glue job as the first step and proceed to the Redshift loading only after the Glue job completes successfully.
- B. Incorrect.
This option is incorrect because S3 event notifications are not suited for scheduling jobs at specific times (e.g., midnight). They are triggered based on object-related events in the bucket, which does not align with the scenario.
- C. Incorrect.
This option is incorrect because AWS Glue Workflows do not provide a native feature to load data into Redshift. Additionally, AWS Batch is not an appropriate service for this scenario since it is designed for batch computing jobs rather than orchestrating data pipelines.
- D. Incorrect.
This option is incorrect because AWS Data Pipeline, while capable of scheduling and orchestration, is not the recommended service for modern, serverless workflows. AWS Step Functions and EventBridge are better suited for this use case.