DEA-C01 Question 97
Select 2You are tasked with building a data pipeline on AWS that ingests data from a source system daily at 7 AM, processes it through a series of transformations, and then loads the processed data into an Amazon Redshift cluster. The pipeline must ensure that the second step (data transformations) only begins after the first step (data ingestion) is completed successfully. Which combination of AWS services and features should you use to achieve this?
- A
Use AWS Step Functions to orchestrate the pipeline, setting the ingestion and transformation steps as sequential states.
- B
Use Amazon EventBridge to trigger an AWS Lambda function for data ingestion at 7 AM and another Lambda function for data transformation.
- C
Use AWS Glue workflows to define the dependencies between the ingestion and transformation steps.
- D
Use Amazon Kinesis Data Firehose to ingest data into Amazon Redshift and configure a schedule for daily processing.
- E
Use Amazon S3 with lifecycle policies to manage the order of the ingestion and transformation steps.
Show answer and explanation
Correct answers: A, C
Explanation
To configure a data pipeline with dependencies, you need services that can orchestrate workflows and manage dependencies between steps. AWS Step Functions is a serverless orchestration service that allows you to define complex workflows, including sequential execution. AWS Glue workflows are specifically designed for ETL tasks and also support dependency management. EventBridge and Kinesis Data Firehose are useful for triggering and streaming but do not handle step dependencies, and S3 lifecycle policies are unrelated to pipeline orchestration.
- A. Correct.
This is correct. AWS Step Functions allows you to orchestrate workflows with dependencies, ensuring that each step runs only after the previous one is completed successfully.
- B. Incorrect.
This is incorrect. While Amazon EventBridge can schedule tasks, it does not inherently manage dependencies between steps in a data pipeline.
- C. Correct.
This is correct. AWS Glue workflows provide a way to define, manage, and visualize complex ETL workflows, including dependencies between steps.
- D. Incorrect.
This is incorrect. Amazon Kinesis Data Firehose is used for streaming data into destinations like Amazon Redshift but does not inherently support orchestration of steps or scheduling.
- E. Incorrect.
This is incorrect. Amazon S3 lifecycle policies are used for managing data retention and storage transitions, not for orchestrating or scheduling workflows.