DEA-C01 Question 388
Select 2You are designing a data pipeline in AWS to process large amounts of streaming data from IoT devices in real-time. The data needs to be ingested, processed, and stored for analytics. The processing must support transformations and aggregations with low latency. Which combination of services would best meet these requirements?
- A
Amazon Kinesis Data Streams for ingestion, AWS Lambda for processing, and Amazon S3 for storage
- B
Amazon Kinesis Data Firehose for ingestion and processing, and Amazon Redshift for storage
- C
Amazon Kinesis Data Streams for ingestion, Amazon Kinesis Data Analytics for processing, and Amazon S3 for storage
- D
Amazon SQS for ingestion, AWS Glue for processing, and Amazon DynamoDB for storage
- E
Amazon MSK (Managed Streaming for Apache Kafka) for ingestion, Apache Flink on Amazon EMR for processing, and Amazon S3 for storage
Show answer and explanation
Correct answers: A, C
Explanation
For a real-time data pipeline, Amazon Kinesis Data Streams is an excellent choice for ingesting streaming data at scale. AWS Lambda and Amazon Kinesis Data Analytics provide powerful, low-latency processing options for transformations and aggregations. Amazon S3 is a cost-effective and durable storage solution for the processed data. These services work seamlessly together to meet the requirements of scalability, low latency, and real-time processing.
- A. Correct.
Correct: Amazon Kinesis Data Streams is ideal for real-time ingestion of streaming data, AWS Lambda can process events with low latency, and Amazon S3 provides scalable and cost-effective storage for processed data.
- B. Incorrect.
Incorrect: While Amazon Kinesis Data Firehose supports ingestion and basic processing, it lacks advanced real-time processing capabilities like transformations and aggregations. Amazon Redshift is more suited for analytical queries, not for scalable real-time data storage.
- C. Correct.
Correct: Amazon Kinesis Data Streams supports real-time ingestion, Amazon Kinesis Data Analytics allows for in-stream transformations and aggregations with low latency, and Amazon S3 provides a durable and cost-effective storage solution.
- D. Incorrect.
Incorrect: Amazon SQS is not designed for real-time ingestion of high-throughput streaming data. AWS Glue is more suited for batch ETL processing rather than low-latency real-time data transformations. Amazon DynamoDB is a NoSQL database, not designed for analytics-focused storage.
- E. Incorrect.
Incorrect: While Amazon MSK and Apache Flink support real-time ingestion and processing, Apache Flink on Amazon EMR introduces additional operational overhead compared to managed services like Amazon Kinesis Data Analytics. Amazon S3 is appropriate for storage, but this combination is less efficient for the outlined requirements.