DEA-C01 Question 23
Select 2You are a data engineer tasked with implementing a solution for real-time streaming data ingestion from IoT devices. The data must be ingested with low latency and subsequently processed in near real-time for analytics. The solution should also handle variable message throughput efficiently. Which combination of AWS services would most effectively meet these requirements?
- A
Amazon Kinesis Data Streams for ingestion and AWS Lambda for processing
- B
Amazon S3 for ingestion and Amazon Athena for processing
- C
Amazon MSK (Managed Streaming for Apache Kafka) for ingestion and AWS Glue for processing
- D
Amazon Kinesis Data Firehose for ingestion and Amazon Redshift for processing
- E
Amazon SQS for ingestion and Amazon EMR for processing
Show answer and explanation
Correct answers: A, C
Explanation
For real-time streaming data ingestion and processing, it is essential to use services that support low-latency and high-throughput scenarios. Amazon Kinesis Data Streams and Amazon MSK are specifically designed for streaming data ingestion with variable message throughput. AWS Lambda and AWS Glue are suitable for processing the data in near real-time, depending on the specific use case. Other options, like Amazon S3 or SQS, are not optimized for real-time ingestion, and services like Athena or EMR are better suited for batch processing rather than real-time analytics.
- A. Correct.
Amazon Kinesis Data Streams is designed for low-latency streaming data ingestion, and AWS Lambda can process the ingested data in near real-time. This combination is ideal for IoT use cases with variable message throughput.
- B. Incorrect.
Amazon S3 is a storage service rather than a streaming data ingestion service. While Amazon Athena can process data stored in S3, it is not well-suited for real-time analytics as it operates on batch data.
- C. Correct.
Amazon MSK provides a managed Kafka service for high-throughput, low-latency streaming data ingestion. AWS Glue can be used for near real-time ETL processing of the data.
- D. Incorrect.
Amazon Kinesis Data Firehose is primarily used for delivering data to storage destinations like S3, Redshift, or Elasticsearch, but it does not provide real-time data processing capabilities.
- E. Incorrect.
Amazon SQS is a message queueing service, not optimized for streaming data ingestion. While Amazon EMR can handle large-scale data processing, it is better suited for batch processing rather than real-time analytics.