MLS-C01 Question 27
Select 2You are designing a machine learning pipeline that involves both batch processing and real-time data streaming for a retail application. The pipeline must ingest transaction data in real-time for fraud detection while also processing historical sales data nightly to train new ML models. Which set of AWS services should you use to orchestrate these workloads effectively?
- A
Amazon Kinesis Data Streams for real-time ingestion and Amazon Glue for batch processing
- B
AWS Step Functions to orchestrate batch processing and Amazon S3 for historical data storage
- C
Amazon SQS for real-time ingestion and Amazon EMR for batch processing
- D
Amazon Managed Streaming for Apache Kafka (Amazon MSK) for real-time ingestion and AWS Glue for batch processing
- E
AWS Lambda for real-time ingestion and Amazon SageMaker Processing for batch processing
Show answer and explanation
Correct answers: A, D
Explanation
To orchestrate a pipeline with both batch and streaming workloads, you need services specialized for each use case. For real-time ingestion, Amazon Kinesis Data Streams or Amazon MSK are ideal options as they handle high-throughput streaming data efficiently. For batch processing of historical data, AWS Glue is a managed ETL service well-suited for transforming and preparing datasets. This combination ensures that both streaming and batch requirements are met effectively.
- A. Correct.
Correct. Amazon Kinesis Data Streams is designed for real-time data ingestion, making it suitable for streaming workloads. Amazon Glue is a fully managed ETL service that is effective for batch processing of historical data.
- B. Incorrect.
Incorrect. While AWS Step Functions can orchestrate workflows, it is not used directly for data ingestion or processing. Amazon S3 is primarily for storage and doesn't handle ingestion or batch processing.
- C. Incorrect.
Incorrect. Amazon SQS is a message queuing service, not a real-time streaming service. While Amazon EMR can process batch data, this option does not address the real-time streaming requirement.
- D. Correct.
Correct. Amazon MSK is a managed service for Apache Kafka, which is ideal for real-time streaming ingestion. AWS Glue is a suitable choice for batch processing of historical data.
- E. Incorrect.
Incorrect. While AWS Lambda can handle real-time event-driven workloads, it is not ideal for high-throughput real-time ingestion like Kinesis or MSK. Amazon SageMaker Processing is not designed for orchestrating batch ETL jobs.