DEA-C01 Question 45
Single answerA company processes large volumes of clickstream data in real-time to generate user behavior insights. The data arrives in Amazon Kinesis Data Streams and needs to be transformed before loading it into an Amazon Redshift table for analytics. The transformation includes cleaning the data, filtering out invalid records, and aggregating user activity over a 5-minute window. Which AWS service or combination of services should you use to efficiently transform and process the data?
- A
Use AWS Glue to clean and transform the data, followed by AWS Lambda to aggregate the data in 5-minute windows.
- B
Use Amazon Kinesis Data Analytics to clean, filter, and aggregate the data in real-time.
- C
Use Amazon EMR with Apache Spark to clean, filter, and aggregate the data in batches.
- D
Use AWS Batch for data cleaning and transformation, and Amazon Athena for aggregation.
Show answer and explanation
Correct answer: B
Explanation
In this scenario, the data needs to be processed in real-time, which includes cleaning, filtering, and aggregating the stream over a 5-minute window. Amazon Kinesis Data Analytics is specifically designed for real-time stream processing and supports SQL-like queries for transformations and aggregations. This makes it the most appropriate service for the given requirements.
- A. Incorrect.
AWS Glue is primarily used for ETL (Extract, Transform, Load) operations on data stored in S3 or cataloged in the Glue Data Catalog. AWS Lambda is not well-suited for windowed aggregations over real-time streaming data.
- B. Correct.
Amazon Kinesis Data Analytics is designed specifically for real-time data processing and can perform streaming transformations, including cleaning, filtering, and windowed aggregations, making it the most efficient choice in this scenario.
- C. Incorrect.
Amazon EMR with Apache Spark is a great option for batch processing, but it is not optimized for real-time transformations and windowed aggregations of streaming data.
- D. Incorrect.
AWS Batch is designed for batch job execution and is not appropriate for real-time data transformations. Similarly, Amazon Athena is a query service and does not perform data transformations or real-time processing.