DEA-C01 Question 35
Select 3You are designing a data ingestion pipeline for a financial application that processes transaction data in real-time. To ensure data consistency and reliability, you aim to make the pipeline replayable in case of failures or data corruption. Which of the following design approaches will help you achieve replayability in your pipeline?
- A
Store raw incoming data in Amazon S3 with versioning enabled before processing.
- B
Implement exactly-once processing logic in the stream processing layer using Amazon Kinesis Data Analytics.
- C
Use Amazon SQS FIFO queues to preserve the order of events and deduplicate messages during retries.
- D
Process data directly from the streaming source without storing intermediate states.
- E
Log the offsets or checkpoint data of processed records in a durable and consistent storage like Amazon DynamoDB.
Show answer and explanation
Correct answers: A, C, E
Explanation
Replayability in a data ingestion pipeline ensures you can recover from failures or reprocess data when required. Key strategies include persisting raw data (e.g., in Amazon S3), maintaining message order and deduplication (e.g., with Amazon SQS FIFO queues), and storing processing state (e.g., offsets or checkpoints in Amazon DynamoDB). These techniques collectively help to create a robust pipeline that can handle reprocessing needs in case of unexpected issues.
- A. Correct.
Storing raw data in Amazon S3 with versioning enables you to replay the original data in case of failures, as the unprocessed data is preserved and easily accessible.
- B. Incorrect.
While exactly-once processing is critical for correctness, it does not inherently ensure replayability of the pipeline. Replayability requires the ability to reprocess original data if needed.
- C. Correct.
Using Amazon SQS FIFO queues ensures that message order is maintained and duplicates are eliminated, enabling easier replay or retry mechanisms.
- D. Incorrect.
Processing data directly from the streaming source without intermediate storage makes replayability difficult, as there is no persistent backup of the raw data for reprocessing in case of failures.
- E. Correct.
Logging offsets or checkpoint data in a durable store like Amazon DynamoDB ensures that you can resume processing from the last known state, enabling partial replays and recovery.