DEA-C01 Question 99
Select 4You are designing a data pipeline on AWS to process log data daily and generate a report. The pipeline involves extracting data from Amazon S3, transforming it using AWS Glue, and loading the processed data into Amazon Redshift. The pipeline must be triggered daily at midnight, and the Glue job should not start until the S3 data extraction is complete. How should you configure the pipeline to meet these requirements?
- A
Use Amazon EventBridge to schedule the pipeline to start daily at midnight and configure AWS Glue job triggers with a dependency on the data extraction step.
- B
Use AWS Step Functions to orchestrate the workflow, ensuring the Glue job starts only after the data extraction step is completed.
- C
Use Amazon Managed Workflows for Apache Airflow (MWAA) to define a Directed Acyclic Graph (DAG) for the pipeline, with dependencies defined between the S3 data extraction and Glue job.
- D
Use AWS Data Pipeline to define the schedule and task dependencies for the S3 data extraction and AWS Glue job steps.
- E
Use AWS Lambda to trigger the pipeline daily and include a delay function to ensure the Glue job starts after the data extraction step.
Show answer and explanation
Correct answers: A, B, C, D
Explanation
The required pipeline involves scheduling and dependency management. Amazon EventBridge, AWS Step Functions, Amazon MWAA, and AWS Data Pipeline are all valid solutions for achieving this. EventBridge can handle the scheduling, and Glue triggers can enforce the dependency. Step Functions and MWAA allow for defining workflows with dependencies, while AWS Data Pipeline is specifically built for such use cases. AWS Lambda, while useful for certain tasks, is not a robust or recommended solution for orchestrating pipelines with dependencies.
- A. Correct.
Using Amazon EventBridge for scheduling and AWS Glue triggers for dependencies is a valid solution. EventBridge can trigger the pipeline daily at midnight, and Glue job triggers can enforce dependencies between steps.
- B. Correct.
AWS Step Functions is a state machine service designed for orchestrating workflows. It allows you to define the pipeline steps with clear dependencies, ensuring the Glue job starts only after the data extraction step.
- C. Correct.
Amazon MWAA can manage complex workflows using DAGs. You can create a DAG to define the pipeline's tasks and their dependencies, ensuring the Glue job starts after the S3 data extraction.
- D. Correct.
AWS Data Pipeline is specifically designed for creating scheduled workflows with task dependencies. You can use it to orchestrate the S3 data extraction and Glue job, meeting the requirements.
- E. Incorrect.
Using AWS Lambda with a delay is not a reliable solution for managing dependencies in a scheduled pipeline. It does not provide robust dependency management and can lead to unpredictable behavior in case of delays or failures.