DEA-C01 Question 6
Select 2You are designing a data pipeline using AWS services to process real-time streaming data from IoT devices. The pipeline must ensure exactly-once processing and be able to handle sudden spikes in data volume. Which combination of AWS services should you use to meet these requirements?
- A
Amazon Kinesis Data Streams for ingestion and Amazon Kinesis Data Analytics for processing
- B
Amazon Managed Streaming for Apache Kafka (Amazon MSK) for ingestion and Apache Flink for processing
- C
Amazon SQS for ingestion and AWS Lambda for processing
- D
Amazon Kinesis Data Streams for ingestion and AWS Glue for processing
- E
Amazon EventBridge for ingestion and AWS Step Functions for processing
Show answer and explanation
Correct answers: A, B
Explanation
To process real-time streaming data with exactly-once semantics and handle sudden spikes in data volume, you need a combination of services optimized for streaming ingestion and processing. Both Amazon Kinesis Data Streams with Kinesis Data Analytics and Amazon MSK with Apache Flink meet these requirements. Amazon SQS, AWS Glue, and Amazon EventBridge are not suitable for real-time streaming or ensuring exactly-once processing in this context.
- A. Correct.
Amazon Kinesis Data Streams provides a durable and scalable solution for ingesting real-time streaming data. Amazon Kinesis Data Analytics enables real-time processing and can ensure exactly-once processing if properly configured.
- B. Correct.
Amazon MSK is a managed service for Apache Kafka, which is well-suited for real-time streaming ingestion and can handle high data volumes. Apache Flink is designed for stateful stream processing and supports exactly-once semantics.
- C. Incorrect.
Amazon SQS is typically used for queue-based decoupling, but it is not optimized for real-time streaming ingestion or exactly-once semantics in a streaming context.
- D. Incorrect.
While Amazon Kinesis Data Streams is a good choice for ingestion, AWS Glue is primarily designed for batch ETL processes, not real-time streaming processing.
- E. Incorrect.
Amazon EventBridge is designed for event-driven architectures, not high-throughput real-time streaming. AWS Step Functions can orchestrate workflows but is not suited for real-time stream processing.