MLA-C01 Question 338
Select 3You are tasked with automating the ingestion and preprocessing of batch data from multiple on-premises sources into an Amazon S3 data lake. The workflow must include data validation, transformation, and logging of errors. Which combination of services would you use to design an end-to-end solution with minimal operational overhead?
- A
AWS Glue for data transformation and Amazon S3 for storage
- B
AWS Step Functions for orchestration and Amazon SNS for error notifications
- C
AWS DataSync for data transfer and AWS Lambda for validation
- D
Amazon S3 Event Notifications to trigger AWS Batch for transformation tasks
- E
Amazon Kinesis Data Streams for real-time ingestion and processing
Show answer and explanation
Correct answers: A, B, C
Explanation
To automate and integrate batch data ingestion, validation, transformation, and error handling, a combination of services is required. AWS Glue can transform data, AWS DataSync simplifies transferring on-premises data to Amazon S3, and AWS Lambda can validate the data. AWS Step Functions orchestrates the workflow, and Amazon SNS handles notifications for error reporting. This setup minimizes operational overhead and ensures an end-to-end automated solution.
- A. Correct.
Correct: AWS Glue can handle data transformation, and Amazon S3 provides a scalable storage solution for the data lake.
- B. Correct.
Correct: AWS Step Functions can orchestrate complex workflows, and Amazon SNS can send notifications for errors or exceptions.
- C. Correct.
Correct: AWS DataSync simplifies the transfer of on-premises data to the cloud, and AWS Lambda can validate the data during ingestion.
- D. Incorrect.
Incorrect: Amazon S3 Event Notifications can trigger some actions, but AWS Batch is not well-suited for this use case as it is designed for batch processing rather than orchestrating multi-step workflows.
- E. Incorrect.
Incorrect: Amazon Kinesis Data Streams is designed for real-time ingestion, which is not required in this case as the scenario involves batch data.