MLS-C01 Question 25
Select 2You are building a machine learning pipeline that processes both batch and real-time data for a retail company. The batch data includes nightly inventory updates stored in Amazon S3, while the real-time data consists of customer transactions streaming from point-of-sale (POS) systems. You need to design an orchestration pipeline that ingests both types of data and ensures the data is available for model training and inference. Which combination of AWS services would best fulfill these requirements?
- A
AWS Glue for batch data processing and Amazon Kinesis Data Streams for real-time data ingestion
- B
Amazon EMR for batch data processing and Amazon SNS for real-time notification of streaming data
- C
AWS Glue for batch data processing and Amazon Kinesis Data Firehose for real-time data delivery
- D
Amazon S3 for batch data storage and AWS Lambda for real-time data processing
- E
Amazon QuickSight for data visualization and Amazon SageMaker for real-time inference
Show answer and explanation
Correct answers: A, C
Explanation
To orchestrate data ingestion pipelines for both batch and real-time ML workloads, AWS Glue and Amazon Kinesis services are highly effective. AWS Glue is designed for batch ETL jobs, while Amazon Kinesis Data Streams and Kinesis Data Firehose are purpose-built for real-time streaming data ingestion and delivery. This combination ensures data is ingested, transformed, and made available for downstream ML workflows efficiently.
- A. Correct.
Correct: AWS Glue is well-suited for batch data processing and transformations, while Amazon Kinesis Data Streams can handle real-time data ingestion with low latency, making this an ideal combination for the scenario.
- B. Incorrect.
Incorrect: While Amazon EMR is suitable for large-scale batch processing, Amazon SNS is a messaging service and not designed for real-time data ingestion or processing. This does not fulfill the requirements for managing both batch and streaming data.
- C. Correct.
Correct: AWS Glue can handle ETL for batch processing, and Amazon Kinesis Data Firehose is an excellent choice for reliably delivering streaming data to storage destinations like Amazon S3. This combination meets the requirements for both batch and streaming data ingestion.
- D. Incorrect.
Incorrect: Amazon S3 is a storage service and does not handle batch processing. AWS Lambda can process real-time events but is not designed for high-throughput streaming data ingestion like Kinesis.
- E. Incorrect.
Incorrect: Amazon QuickSight is a business intelligence tool for visualization, and Amazon SageMaker is for building, training, and deploying models. Neither service addresses batch or streaming data ingestion and processing specifically.