DEA-C01 Question 50
Select 2You are designing a data pipeline on AWS to process streaming data from IoT sensors. The data needs to be ingested in near real-time, processed for anomalies, and stored for long-term analysis. Which combination of services should you use to meet these requirements efficiently?
- A
Amazon Kinesis Data Streams for ingestion, AWS Lambda for processing, and Amazon S3 for storage
- B
Amazon SQS for ingestion, AWS Glue for processing, and Amazon Redshift for storage
- C
Amazon Kinesis Firehose for ingestion, AWS Lambda for processing, and Amazon S3 for storage
- D
Amazon Kinesis Data Streams for ingestion, Amazon EMR for processing, and Amazon DynamoDB for storage
- E
Amazon Kinesis Data Streams for ingestion, Amazon Managed Streaming for Apache Kafka (MSK) for processing, and Amazon S3 for storage
Show answer and explanation
Correct answers: A, C
Explanation
When building a streaming data pipeline, Amazon Kinesis services are well-suited for real-time ingestion. AWS Lambda provides a serverless method to process and analyze incoming data for anomalies. Amazon S3 is a cost-efficient and scalable storage solution for long-term data retention. Both Option 1 and Option 3 use these services in effective combinations to meet the requirements of real-time processing and long-term storage efficiently.
- A. Correct.
Correct: Amazon Kinesis Data Streams is ideal for ingesting streaming data in near real-time, AWS Lambda can process data for anomalies in a serverless manner, and Amazon S3 is a cost-effective solution for long-term storage.
- B. Incorrect.
Incorrect: Amazon SQS is a queue service, not optimal for real-time streaming. AWS Glue is more suited for batch ETL processing, not real-time anomaly detection.
- C. Correct.
Correct: Amazon Kinesis Firehose is suitable for real-time ingestion and data delivery to destinations like Amazon S3. AWS Lambda can efficiently process the data, and Amazon S3 is a reliable storage option.
- D. Incorrect.
Incorrect: While Amazon EMR can process large-scale data, it is more suitable for big data batch processing than near real-time processing. Additionally, Amazon DynamoDB is not an ideal choice for long-term storage of large datasets.
- E. Incorrect.
Incorrect: Amazon MSK is a managed Kafka service that could work for processing, but it is more complex to set up and manage compared to Lambda for real-time anomaly detection. Furthermore, this option does not cover anomaly detection adequately.