MLA-C01 Question 336
Select 2You are building a machine learning pipeline to process customer transaction data daily. The data resides in an S3 bucket and must be transformed and validated before being saved to a data warehouse in Amazon Redshift. The pipeline must automate the ingestion, transformation, and loading process while handling errors gracefully. Which combination of AWS services should you use to orchestrate and automate this pipeline?
- A
AWS Glue for data transformation and Amazon S3 event notifications to trigger the pipeline
- B
AWS Step Functions to orchestrate the pipeline and AWS Lambda for custom error handling
- C
Amazon Kinesis Data Streams to ingest the data and Amazon EMR for transformation
- D
AWS Glue Workflow to orchestrate the pipeline with built-in error handling
- E
Amazon SageMaker Processing Jobs for data transformation and AWS Lambda for orchestration
Show answer and explanation
Correct answers: A, B
Explanation
To automate and integrate the data ingestion pipeline, you need services that can handle orchestration, error handling, and data transformations. AWS Glue provides a serverless ETL solution for data transformation, while Amazon S3 event notifications can trigger the pipeline when new data arrives. AWS Step Functions, on the other hand, are well-suited for orchestrating workflows involving multiple services and ensuring error handling, making the combination of AWS Glue, S3 event notifications, and Step Functions an optimal solution.
- A. Correct.
Correct: AWS Glue can be used for data transformation, and Amazon S3 event notifications can trigger the pipeline whenever new data is uploaded to the bucket, ensuring automation and integration.
- B. Correct.
Correct: AWS Step Functions provide a way to orchestrate complex workflows and handle errors gracefully, while AWS Lambda can be used to implement custom error handling logic for the pipeline.
- C. Incorrect.
Incorrect: While Amazon Kinesis Data Streams is suitable for real-time data ingestion, this scenario involves daily batch processing, making it less optimal than other solutions.
- D. Incorrect.
Incorrect: AWS Glue Workflow can orchestrate Glue jobs, but it lacks the flexibility and error-handling capabilities of AWS Step Functions for this scenario.
- E. Incorrect.
Incorrect: Amazon SageMaker Processing Jobs are designed for large-scale data processing in the context of machine learning workflows and are not ideal for orchestrating a data pipeline. AWS Lambda is also not suitable for full orchestration of such a pipeline.