DEA-C01 Question 22
Select 3You are tasked with designing a real-time data ingestion pipeline for a stock trading application that processes millions of transactions per second. The application requires low-latency processing and the ability to replay data in case of failures. Which combination of services should you use to achieve this?
- A
Amazon Kinesis Data Streams for ingestion and AWS Lambda for processing
- B
Amazon S3 for ingestion and Amazon Athena for querying
- C
Amazon Kinesis Data Streams for ingestion and Amazon Kinesis Data Analytics for real-time processing
- D
Amazon DynamoDB Streams for ingestion and AWS Glue for processing
- E
Amazon Managed Streaming for Apache Kafka (Amazon MSK) for ingestion and Apache Flink for processing
Show answer and explanation
Correct answers: A, C, E
Explanation
For real-time, low-latency data ingestion and processing, services like Amazon Kinesis Data Streams and Amazon MSK are ideal for handling high-throughput streaming data. They also support replaying records, which meets the application's requirements. AWS Lambda, Amazon Kinesis Data Analytics, and Apache Flink are well-suited for real-time processing tasks. Other options, such as Amazon S3 and DynamoDB Streams, are not designed for low-latency, real-time scenarios, making them unsuitable for this use case.
- A. Correct.
Amazon Kinesis Data Streams is designed for high-throughput, low-latency streaming data ingestion, and AWS Lambda can process the data in real-time. This combination is suitable for real-time processing and replaying data.
- B. Incorrect.
Amazon S3 and Amazon Athena are not suitable for low-latency, real-time use cases as they are geared towards batch processing and querying historical data.
- C. Correct.
Amazon Kinesis Data Streams provides low-latency ingestion, and Amazon Kinesis Data Analytics can directly process streaming data in real-time. This is an effective combination for the given use case.
- D. Incorrect.
Amazon DynamoDB Streams can capture item-level changes in DynamoDB, but it is not designed for high-throughput streaming ingestion from external sources, making it unsuitable for this use case.
- E. Correct.
Amazon MSK is a managed service for Apache Kafka, which is highly effective for low-latency, high-throughput streaming ingestion. Apache Flink is a robust framework for real-time stream processing, making this combination suitable for the scenario.