DEA-C01 Question 79
Select 3You are working as a Data Engineer for a company that processes large amounts of clickstream data. You need to build an end-to-end data pipeline that ingests data from Amazon Kinesis Data Streams, processes it using AWS Glue, and stores the processed results in Amazon Redshift for analytics. Additionally, the pipeline must be scheduled to run daily and should provide monitoring for failures. Which combination of AWS services and features should you use to orchestrate this pipeline?
- A
Use AWS Step Functions to orchestrate the pipeline, defining tasks for each step and error handling logic.
- B
Use Amazon EventBridge to schedule the pipeline to run daily and trigger an AWS Lambda function.
- C
Use AWS Glue Workflows to organize the pipeline steps and monitor their execution.
- D
Use Amazon CloudWatch Alarms to monitor the pipeline for failures and trigger notifications.
- E
Use Amazon S3 Event Notifications to trigger the start of the pipeline whenever new data arrives in S3.
Show answer and explanation
Correct answers: A, C, D
Explanation
To orchestrate a data pipeline with multiple steps, including ingestion, processing, and storage, you need a combination of services that handle orchestration, scheduling, and monitoring. AWS Step Functions and AWS Glue Workflows are ideal for defining and managing the pipeline steps, while Amazon CloudWatch Alarms ensures that failures are detected and notified. EventBridge and S3 Event Notifications, while useful in other scenarios, are not the best fit for this specific use case.
- A. Correct.
Correct: AWS Step Functions can be used to orchestrate a pipeline by defining tasks and their dependencies, including error handling and retry logic.
- B. Incorrect.
Incorrect: While Amazon EventBridge can schedule tasks, it is not specifically designed for orchestrating complex pipelines with multiple steps and dependencies.
- C. Correct.
Correct: AWS Glue Workflows is a native AWS Glue feature designed for organizing and monitoring data pipelines, making it a suitable choice for this scenario.
- D. Correct.
Correct: Amazon CloudWatch Alarms can monitor the pipeline for failures by tracking metrics or logs and can be configured to send notifications when issues occur.
- E. Incorrect.
Incorrect: Amazon S3 Event Notifications are used to trigger events when new data arrives in S3, but in this scenario, the pipeline starts with Kinesis Data Streams, not S3.