MLS-C01 Question 28
Select 2You are building a data ingestion pipeline for a machine learning workload that processes both batch and streaming data from IoT sensors. The batch data is uploaded to Amazon S3 every hour, while the streaming data arrives in real time. The pipeline must ensure low-latency ingestion for the streaming data and handle the batch data efficiently. Which combination of services should you use to orchestrate this pipeline?
- A
Use Amazon Kinesis Data Streams for streaming data ingestion and AWS Glue for batch data processing.
- B
Use AWS Lambda to process streaming data in real time and Amazon EMR for batch data processing.
- C
Use Amazon SQS for streaming data ingestion and AWS Batch for batch data processing.
- D
Use Amazon Kinesis Data Firehose for streaming data ingestion and AWS Glue for batch data processing.
- E
Use Amazon Redshift for both streaming and batch data ingestion.
Show answer and explanation
Correct answers: A, D
Explanation
To orchestrate a data ingestion pipeline for both batch and streaming ML workloads, you need services optimized for their respective tasks. Amazon Kinesis Data Streams or Amazon Kinesis Data Firehose are excellent choices for streaming data ingestion because they are designed for low-latency, real-time data processing. AWS Glue is a serverless, scalable ETL service that is ideal for batch data processing and can integrate with Amazon S3. Using these services together ensures an efficient and scalable pipeline for both batch and streaming data.
- A. Correct.
Correct: Amazon Kinesis Data Streams is designed for high-throughput real-time streaming data ingestion. AWS Glue can handle ETL tasks for batch data efficiently.
- B. Incorrect.
Partially correct: AWS Lambda can process streaming data in real time, but it is not the most optimal choice for high-throughput streaming data. Amazon EMR is a good choice for batch processing but more suited for large-scale distributed processing.
- C. Incorrect.
Incorrect: Amazon SQS is a message queue service, not designed for high-throughput streaming data ingestion. AWS Batch is not suitable for orchestrating batch ETL tasks.
- D. Correct.
Correct: Amazon Kinesis Data Firehose is a fully managed service for streaming data ingestion and delivery. AWS Glue is well-suited for batch data processing and ETL.
- E. Incorrect.
Incorrect: Amazon Redshift is a data warehousing service and is not intended for real-time streaming data ingestion or batch data orchestration.