DEA-C01 Question 36
Single answerYou are designing a data processing pipeline in AWS to process streaming data from IoT sensors. The pipeline must aggregate sensor data over a 5-minute window and store the aggregated results in an Amazon Redshift table for further analysis. The solution must also ensure that the pipeline can tolerate worker node failures without losing processing state. Which AWS service combination is the most appropriate for this use case?
- A
Amazon Kinesis Data Analytics with checkpointing enabled and Amazon Redshift
- B
Amazon Kinesis Data Streams with AWS Lambda and Amazon Redshift
- C
AWS Glue with Amazon S3 and Amazon Redshift
- D
Amazon EMR with Apache Spark Streaming and Amazon Redshift
Show answer and explanation
Correct answer: A
Explanation
Stateful data processing requires maintaining state across events, which is essential for aggregating data over time windows like the 5-minute window in this scenario. Amazon Kinesis Data Analytics is a fully managed service that provides native support for stateful processing, including checkpointing for fault tolerance. This makes it the optimal choice for handling streaming IoT sensor data and integrating with Amazon Redshift for storing aggregated results.
- A. Correct.
Amazon Kinesis Data Analytics supports stateful processing using checkpointing, which helps maintain processing state in case of worker node failures. It also integrates seamlessly with Amazon Redshift for storing aggregated results.
- B. Incorrect.
While AWS Lambda can process streaming data from Kinesis Data Streams, it does not natively support stateful processing with checkpointing, which is critical for this use case.
- C. Incorrect.
AWS Glue is primarily used for ETL jobs and does not natively support stateful streaming data processing, making it unsuitable for this real-time aggregation requirement.
- D. Incorrect.
Amazon EMR with Apache Spark Streaming can handle stateful processing, but it requires significant configuration and management effort compared to the fully managed capabilities of Kinesis Data Analytics.