DEA-C01 Question 24
Select 2You are tasked with designing a streaming data ingestion pipeline for an application that processes sensor data in real-time. The data is ingested at a high velocity, and you need to ensure low latency and fault tolerance. Which combination of AWS services would be the most suitable for this use case?
- A
Amazon Kinesis Data Streams for ingesting data and AWS Lambda for processing the stream in real-time
- B
Amazon S3 for direct ingestion of streaming data and AWS Glue for real-time transformations
- C
Amazon MSK (Managed Streaming for Apache Kafka) for ingesting data and Amazon EC2 for running Kafka consumers to process the data
- D
Amazon Kinesis Data Firehose for ingesting data and delivering it to Amazon S3, followed by AWS Glue for batch processing
- E
Amazon DynamoDB Streams for ingestion and Amazon EMR for real-time data processing
Show answer and explanation
Correct answers: A, C
Explanation
For a high-velocity, real-time streaming data ingestion pipeline, Amazon Kinesis Data Streams and AWS Lambda provide an ideal combination with low latency, scalability, and fault tolerance. Alternatively, using Amazon MSK for streaming ingestion and Amazon EC2 for processing provides flexibility and scalability, though it involves managing the infrastructure. Other options, such as Amazon S3 and AWS Glue, are not designed for low-latency real-time use cases, and services like DynamoDB Streams and EMR are unsuitable for this specific high-throughput, real-time scenario.
- A. Correct.
Amazon Kinesis Data Streams is designed for high-throughput, real-time data ingestion, and AWS Lambda can process the data in real-time with low latency. This combination is fault-tolerant and scalable.
- B. Incorrect.
Amazon S3 is not suited for direct ingestion of high-velocity streaming data, as it is primarily a storage service. AWS Glue is typically used for ETL jobs and not for real-time data transformations.
- C. Correct.
Amazon MSK is a managed service for Apache Kafka, which is ideal for real-time data ingestion at scale. Using Amazon EC2 for running Kafka consumers provides flexibility for processing the data, though it requires manual management.
- D. Incorrect.
Amazon Kinesis Data Firehose can ingest data and deliver it to Amazon S3, but it is a near real-time service and not suitable for low-latency, real-time use cases. AWS Glue is better suited for batch processing rather than real-time transformations.
- E. Incorrect.
Amazon DynamoDB Streams is not optimized for high-throughput, low-latency streaming data ingestion, and Amazon EMR is better suited for batch or large-scale distributed processing rather than real-time stream processing.