DEA-C01 Question 222
Select 2A data engineering team is building a real-time data pipeline to process streaming data from IoT devices. The data must be ingested, processed with low latency, and stored in a data lake for further analysis. Which combination of AWS services would best meet these requirements?
- A
Amazon Kinesis Data Streams for ingestion, AWS Lambda for processing, and Amazon S3 for storage
- B
Amazon SQS for ingestion, Amazon EMR for processing, and Amazon RDS for storage
- C
Amazon MSK (Managed Streaming for Apache Kafka) for ingestion, AWS Glue for processing, and Amazon Redshift for storage
- D
Amazon Kinesis Data Firehose for ingestion and transformation, and Amazon S3 for storage
- E
AWS IoT Core for ingestion, AWS Batch for processing, and Amazon DynamoDB for storage
Show answer and explanation
Correct answers: A, D
Explanation
For a real-time data pipeline, Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are ideal for data ingestion. AWS Lambda complements these services by providing low-latency, serverless processing. Amazon S3 is the best choice for storing raw streaming data in a data lake due to its scalability, durability, and cost-effectiveness. The other options either do not support real-time processing or are not suitable for the given use case.
- A. Correct.
Amazon Kinesis Data Streams provides low-latency ingestion for streaming data, AWS Lambda offers serverless processing in real time, and Amazon S3 is a scalable, durable storage option suitable for data lakes. This is a correct combination.
- B. Incorrect.
Amazon SQS is primarily a message queuing service, not optimized for high-throughput streaming data. Amazon EMR is better suited for batch processing rather than low-latency use cases, and Amazon RDS is not commonly used for storing large-scale streaming data.
- C. Incorrect.
Amazon MSK is a valid option for ingestion, but AWS Glue is a batch-oriented ETL service, which is not suitable for real-time low-latency processing. Amazon Redshift is a data warehousing solution, not ideal for storing raw streaming data in a data lake.
- D. Correct.
Amazon Kinesis Data Firehose is a fully managed service for streaming data ingestion and transformation. It can deliver data directly to Amazon S3, which is an ideal storage solution for data lakes. This is a correct combination.
- E. Incorrect.
AWS IoT Core is designed specifically for IoT device data ingestion but not general streaming. AWS Batch is designed for batch processing workloads, not real-time processing, and DynamoDB is a NoSQL database, which is not suitable for large-scale raw data lake storage.