DEA-C01 Question 322
Select 2You are designing a data pipeline to process large volumes of log data from multiple sources in near real-time using AWS services. The processed data needs to be stored in a data lake for analytics, while ensuring scalability, high availability, and minimal operational overhead. Which combination of AWS services would best meet these requirements?
- A
Amazon Kinesis Data Streams for ingesting log data in real-time and Amazon S3 for storing processed data
- B
Amazon Redshift for ingesting log data and storing processed data for analytics
- C
AWS Glue for real-time log ingestion and Amazon Kinesis Data Firehose for storing processed data in Amazon S3
- D
Amazon Kinesis Data Firehose for near real-time log ingestion and Amazon S3 for storing processed data
- E
Amazon EMR for ingesting and processing log data and Amazon S3 for storing processed data
Show answer and explanation
Correct answers: A, D
Explanation
The correct solution for real-time log ingestion with scalability and minimal operational overhead involves using Amazon Kinesis Data Streams or Amazon Kinesis Data Firehose for ingestion and Amazon S3 as the data lake for storage. Kinesis services are specifically designed to handle real-time data streams, while Amazon S3 provides a cost-effective and scalable storage solution for processed data ready for analytics.
- A. Correct.
Amazon Kinesis Data Streams is a scalable service for ingesting real-time data streams, and Amazon S3 is a cost-effective and highly available data lake solution for storing large volumes of processed data. This combination is suitable for the given scenario.
- B. Incorrect.
Amazon Redshift is a data warehouse service designed for complex analytics and not optimized for real-time log ingestion or storing raw data in a data lake.
- C. Incorrect.
AWS Glue is primarily used for ETL (extract, transform, load) tasks and does not support real-time log ingestion. Amazon Kinesis Data Firehose, however, is designed for real-time ingestion and can deliver data to Amazon S3.
- D. Correct.
Amazon Kinesis Data Firehose is a fully managed service for near real-time log ingestion and can directly deliver data to Amazon S3 for storage, making it a good choice for this scenario.
- E. Incorrect.
Amazon EMR is a managed Hadoop framework for big data processing, but it is not optimized for real-time log ingestion. While it can store data in Amazon S3, it introduces additional operational overhead compared to Kinesis.