DEA-C01 Question 83
Select 3You are working as a Data Engineer for a company that collects streaming data from IoT sensors in multiple regions. The data must be ingested in real-time, processed to filter out invalid records, and stored in a durable data lake for further analytics. Which combination of AWS services would best meet these requirements?
- A
Amazon Kinesis Data Streams for ingestion, AWS Lambda for processing, and Amazon S3 for storage
- B
Amazon Simple Queue Service (SQS) for ingestion, Amazon RDS for processing, and Amazon Redshift for storage
- C
AWS Glue for ingestion, Amazon EMR for processing, and Amazon DynamoDB for storage
- D
Amazon Kinesis Data Firehose for ingestion, AWS Lambda for processing, and Amazon S3 for storage
- E
Amazon Managed Streaming for Apache Kafka (MSK) for ingestion, Amazon Kinesis Data Analytics for processing, and Amazon S3 for storage
Show answer and explanation
Correct answers: A, D, E
Explanation
Real-time streaming data pipelines in AWS often use services like Amazon Kinesis Data Streams or Amazon MSK for ingestion due to their ability to handle high-throughput streaming data. AWS Lambda and Amazon Kinesis Data Analytics are suitable for real-time processing, allowing you to filter and transform data as it flows through the pipeline. Amazon S3 is an excellent choice for a durable, scalable data lake where the processed data can be stored for analytics or further processing. The other options involve services not optimized for this specific use case, such as SQS or DynamoDB.
- A. Correct.
Correct: Amazon Kinesis Data Streams is well-suited for real-time ingestion of streaming data, AWS Lambda can process and filter data in real-time, and Amazon S3 provides durable storage for a data lake.
- B. Incorrect.
Incorrect: Amazon SQS is designed for message queuing, not high-throughput real-time streaming. Amazon RDS is not ideal for real-time data processing, and Amazon Redshift is not a suitable data lake solution.
- C. Incorrect.
Incorrect: AWS Glue is primarily used for ETL processes, not real-time ingestion. Amazon EMR is more suited for batch processing, and Amazon DynamoDB is not a durable data lake solution.
- D. Correct.
Correct: Amazon Kinesis Data Firehose simplifies real-time data ingestion and delivery to destinations like Amazon S3. AWS Lambda can process the data in real-time, and Amazon S3 provides scalable, durable storage.
- E. Correct.
Correct: Amazon MSK is a managed Apache Kafka service that can handle real-time ingestion of streaming data. Kinesis Data Analytics can process streaming data in real-time, and Amazon S3 is a durable storage solution for the data lake.