DEA-C01 Question 40
Select 3You are designing a data pipeline using AWS Glue and Amazon Kinesis Data Streams to process and transform a real-time data feed from IoT devices. The pipeline must maintain the state of device readings (e.g., the last reported value for each device) to calculate deltas between consecutive readings. Which of the following approaches will help you implement this requirement?
- A
Use an AWS Glue Streaming ETL job with an Apache Spark structured streaming stateful transformation.
- B
Use Kinesis Data Firehose to deliver the data directly to Amazon S3 for analysis.
- C
Implement an AWS Lambda function with Amazon DynamoDB to track the latest device readings.
- D
Write a Kinesis Data Analytics application using SQL to maintain and update the state of device readings.
- E
Use an Amazon SQS queue to store the device readings for stateful processing.
Show answer and explanation
Correct answers: A, C, D
Explanation
Stateful data processing is required to maintain the history or context of previous events. AWS Glue Streaming ETL jobs and Kinesis Data Analytics both support stateful transformations, which can help maintain and update device state in real-time. Additionally, a custom solution using AWS Lambda and Amazon DynamoDB can also achieve stateful processing by maintaining the latest device readings. On the other hand, services like Kinesis Data Firehose and Amazon SQS are inherently stateless and do not support maintaining state.
- A. Correct.
Correct: AWS Glue Streaming ETL jobs support stateful transformations using Apache Spark. This allows you to maintain and update state data across micro-batches.
- B. Incorrect.
Incorrect: Kinesis Data Firehose is a stateless service and does not provide support for stateful data processing. It is primarily used for data delivery to storage solutions like Amazon S3.
- C. Correct.
Correct: AWS Lambda can be integrated with Amazon DynamoDB to create a custom solution for maintaining state. DynamoDB can store the latest device readings for each device.
- D. Correct.
Correct: Kinesis Data Analytics allows you to use SQL-based stateful processing with features like sliding and tumbling windows, which can help maintain and update device state.
- E. Incorrect.
Incorrect: Amazon SQS is a message queue service primarily designed for decoupling applications. It does not natively support stateful data processing.