DEA-C01 Question 33
Select 3You are designing a data ingestion pipeline for a financial application that processes transaction logs from multiple sources. Due to the critical nature of the data, you need the ability to reprocess data in case of pipeline failures or downstream issues. Which of the following approaches ensures replayability of the data ingestion pipeline in AWS?
- A
Store raw data in Amazon S3 using a write-once-read-many (WORM) pattern before processing
- B
Use Amazon Kinesis Data Streams with a retention period configured to hold data for several days
- C
Implement Amazon DynamoDB to store incoming data with a TTL (time-to-live) mechanism for automatic cleanup
- D
Use AWS Glue to directly process and transform streaming data without intermediate storage
- E
Enable versioning on Amazon S3 buckets to maintain previous iterations of the ingested data
Show answer and explanation
Correct answers: A, B, E
Explanation
Replayability of data ingestion pipelines is crucial for ensuring that data can be reprocessed in response to failures or downstream issues. Storing raw data in Amazon S3 using a WORM pattern and enabling versioning provides a durable and reliable source for replaying data. Additionally, Amazon Kinesis Data Streams with a sufficient retention period allows you to reprocess streaming data within the retention window. These approaches ensure that the original data is preserved and accessible for replay when needed.
- A. Correct.
Storing raw data in Amazon S3 using a WORM pattern ensures that the original, unprocessed data is preserved and can be reprocessed later if needed. This approach is highly reliable for replayability.
- B. Correct.
Amazon Kinesis Data Streams supports configurable retention periods, allowing data to be replayed within the retention window. This is useful for replaying data in case of processing failures.
- C. Incorrect.
While Amazon DynamoDB is a scalable NoSQL database, using TTL for automatic cleanup does not align with replayability requirements because data is deleted after the TTL period.
- D. Incorrect.
Processing streaming data directly in AWS Glue without intermediate storage does not allow for replayability, as there is no mechanism to retain or reprocess the original data in case of failures.
- E. Correct.
Enabling versioning on Amazon S3 buckets ensures that all versions of an object are retained. This can help recover previous versions of the ingested data and support replayability.