DEA-C01 Question 114
Select 3You are working as a Data Engineer for an e-commerce company. The company uses Amazon Kinesis Data Streams to process real-time transaction and user activity data. You need to ensure that the data is processed and stored in Amazon S3 for downstream analytics. The processing task involves aggregating events per user for a 5-minute window. Which combination of services and configurations should you use to meet this requirement efficiently?
- A
Use AWS Lambda to process data from Kinesis Data Streams and write aggregated results to Amazon S3.
- B
Use Amazon Kinesis Data Firehose to consume data from Kinesis Data Streams, perform aggregation using built-in transformations, and store the results in Amazon S3.
- C
Use Amazon EMR with Apache Spark to process data from Kinesis Data Streams and write aggregated results to Amazon S3.
- D
Use Amazon Glue Streaming ETL to process data from Kinesis Data Streams and write aggregated results to Amazon S3.
- E
Use AWS Batch to process data from Kinesis Data Streams and store aggregated results in Amazon S3.
Show answer and explanation
Correct answers: A, C, D
Explanation
To process and aggregate data from Kinesis Data Streams, you need a service that supports stateful operations and can handle streaming data. While AWS Lambda is not suited for time-window-based aggregation, Amazon EMR with Apache Spark and Amazon Glue Streaming ETL are both capable solutions for this use case. They can efficiently handle the real-time data, perform the necessary aggregation, and store the results in Amazon S3. Kinesis Data Firehose and AWS Batch are not suitable because they do not provide the required aggregation capabilities or are designed for batch processing rather than streaming.
- A. Correct.
AWS Lambda is a serverless option for lightweight processing tasks, but it is not ideal for stateful operations like aggregating data over a 5-minute window.
- B. Incorrect.
Amazon Kinesis Data Firehose does not support custom aggregation logic. It is used for simple transformations and direct delivery to destinations like Amazon S3.
- C. Correct.
Amazon EMR with Apache Spark is a good choice for complex data processing tasks, including aggregation over time windows. It offers flexibility and scalability for stateful operations.
- D. Correct.
Amazon Glue Streaming ETL supports real-time processing and aggregation of data from Kinesis Data Streams. It is managed and integrates well with AWS analytics services like Amazon S3.
- E. Incorrect.
AWS Batch is designed for batch processing workloads, not real-time or near-real-time streaming data processing tasks.