DEA-C01 Question 357
Select 2You are a data engineer working for a company that processes large-scale streaming data from IoT sensors deployed across multiple locations. The data needs to be ingested, processed in near real-time, and stored for further analytics. Which combination of AWS services would best meet these requirements?
- A
Amazon Kinesis Data Streams for ingestion, AWS Lambda for processing, and Amazon S3 for storage
- B
Amazon SQS for ingestion, AWS Glue for processing, and Amazon Redshift for storage
- C
Amazon Kinesis Data Firehose for ingestion, Amazon EMR for processing, and Amazon S3 for storage
- D
Amazon MSK (Managed Streaming for Apache Kafka) for ingestion, AWS Lambda for processing, and Amazon DynamoDB for storage
- E
Amazon API Gateway for ingestion, AWS Step Functions for processing, and Amazon RDS for storage
Show answer and explanation
Correct answers: A, C
Explanation
To process large-scale streaming data in near real-time, it is essential to use services optimized for streaming ingestion, real-time processing, and scalable storage. Amazon Kinesis services (Data Streams or Data Firehose) are purpose-built for streaming ingestion, while AWS Lambda or Amazon EMR can be used for real-time or large-scale data processing. Amazon S3 is a cost-effective and scalable solution for storing the processed data for analytics, making the first and third options the correct choices.
- A. Correct.
Amazon Kinesis Data Streams is a fully managed service for ingesting real-time streaming data, AWS Lambda can process this data in near real-time, and Amazon S3 is a cost-effective storage solution for further analytics. This is a correct combination.
- B. Incorrect.
While Amazon SQS is a messaging service, it is not designed for high-throughput real-time streaming ingestion. AWS Glue is better suited for batch ETL jobs rather than real-time processing. Amazon Redshift is a data warehouse, not ideal for direct ingestion of streaming data. Thus, this combination is incorrect.
- C. Correct.
Amazon Kinesis Data Firehose is designed for near-real-time data ingestion and delivery to destinations like Amazon S3. Amazon EMR is well-suited for processing large-scale data using frameworks like Apache Spark, and Amazon S3 is a reliable storage option. This is a correct combination.
- D. Incorrect.
Amazon MSK is a managed service for Apache Kafka, which can handle streaming ingestion. However, AWS Lambda might face limitations with high-throughput processing, and Amazon DynamoDB is not optimized for storing large-scale analytical data. This combination is not ideal.
- E. Incorrect.
Amazon API Gateway is designed for building APIs, not for high-throughput ingestion of streaming data. AWS Step Functions is an orchestration service, not suitable for real-time data processing, and Amazon RDS is a relational database, not optimized for streaming data. This combination is incorrect.