DEA-C01 Question 31
Select 2You are designing a data ingestion pipeline for a streaming application that processes data from IoT sensors. The pipeline uses Amazon Kinesis Data Streams for ingestion and AWS Lambda for processing. To ensure replayability in case of processing failures, which approach should you implement?
- A
Enable data retention in Amazon Kinesis Data Streams and process records using sequence numbers.
- B
Store raw data in Amazon S3 before processing it with AWS Lambda.
- C
Use Amazon DynamoDB to store processed records and deduplicate data on replay.
- D
Configure AWS Lambda to directly delete records from the Kinesis stream after successful processing.
- E
Set up Dead Letter Queues (DLQs) for AWS Lambda to capture failed events.
Show answer and explanation
Correct answers: A, B
Explanation
Replayability in data ingestion pipelines is crucial for recovering from failures or processing errors. Enabling data retention in Amazon Kinesis Data Streams ensures you can replay data by reading specific sequence numbers. Additionally, storing raw data in Amazon S3 provides a durable backup that can be used for reprocessing. Together, these approaches ensure a robust and replayable pipeline. Other options like using DynamoDB or DLQs address related concerns like deduplication or failure handling but do not directly facilitate replayability.
- A. Correct.
Correct: Enabling data retention in Amazon Kinesis Data Streams allows you to replay data by reading records using sequence numbers, which is critical for replayability.
- B. Correct.
Correct: Storing raw data in Amazon S3 ensures that you have a durable and replayable source of truth in case the pipeline needs to reprocess the data.
- C. Incorrect.
Incorrect: While DynamoDB can help with deduplication, it does not directly ensure replayability of the data ingestion pipeline.
- D. Incorrect.
Incorrect: Configuring AWS Lambda to delete records directly from the Kinesis stream after processing could lead to data loss in case of failures, reducing replayability.
- E. Incorrect.
Incorrect: Dead Letter Queues (DLQs) capture failed events but do not inherently enable replayability of the data ingestion pipeline.