DEA-C01 Question 32
Select 3You are designing a data ingestion pipeline on AWS for a financial application. The pipeline must ensure that data can be reprocessed in case of failures or if new transformations are required in the future. Which of the following approaches will help you achieve replayability in your ingestion pipeline?
- A
Store raw data in Amazon S3 before processing it with downstream systems.
- B
Use Amazon Kinesis Data Streams with data retention enabled to buffer incoming data.
- C
Process data in real-time and store only the final transformed results in Amazon Redshift.
- D
Implement idempotent processing logic in your data transformation jobs.
- E
Use AWS Glue to directly extract and transform data without persisting intermediate results.
Show answer and explanation
Correct answers: A, B, D
Explanation
Replayability in data ingestion pipelines is crucial to handle failures, reprocess data, or apply updated transformations. Storing raw data in Amazon S3 provides a durable and cost-effective way to retain the source data. Amazon Kinesis Data Streams enables event replay within the data retention period, which is useful for fixing downstream issues. Additionally, idempotent processing logic ensures that replaying data does not lead to inconsistent results. Together, these approaches establish a robust replayable pipeline.
- A. Correct.
Storing raw data in Amazon S3 ensures a durable and cost-effective way to retain the original data, enabling reprocessing or replay in case of failures or future transformations.
- B. Correct.
Amazon Kinesis Data Streams with data retention allows you to re-read and replay events from a specified window, which is crucial for ensuring replayability in case of downstream issues.
- C. Incorrect.
Storing only final transformed results in Amazon Redshift does not provide replayability since the raw data is not retained for future processing.
- D. Correct.
Idempotent processing ensures that repeated processing of the same data does not result in duplicate or incorrect results, which is critical for replay scenarios.
- E. Incorrect.
Using AWS Glue without persisting intermediate results does not support replayability as the raw or intermediate data is not stored for reprocessing.