DEA-C01 Question 403
Select 2You are a data engineer tasked with designing a data pipeline in AWS to process incoming customer transactions. The source data is stored in Amazon S3 as CSV files, and it is loaded into an Amazon Redshift data warehouse for analytics. You need to ensure the data is validated for completeness, consistency, accuracy, and integrity before it is loaded into Redshift. Which of the following actions should you take as part of the validation process? (Select TWO.)
- A
Use AWS Glue DataBrew to profile the data in S3 and check for missing or null values.
- B
Enable Amazon Redshift Spectrum to automatically validate data consistency during query execution.
- C
Implement Lambda functions to compare checksums of files in S3 with expected checksum values.
- D
Create SQL constraints in Amazon Redshift tables to enforce data integrity rules during loading.
- E
Set up S3 Versioning to ensure historical versions of files are available for validation.
Show answer and explanation
Correct answers: A, C
Explanation
To ensure comprehensive data validation, AWS Glue DataBrew can profile data for completeness and accuracy by identifying issues such as missing or null values. Additionally, implementing Lambda functions to compare file checksums helps maintain data integrity by verifying that the source data has not been altered. These actions collectively address the requirements for validating data completeness, accuracy, and integrity before loading it into Amazon Redshift.
- A. Correct.
Using AWS Glue DataBrew to profile the data helps identify issues like missing or null values, ensuring data completeness and accuracy before loading into Redshift.
- B. Incorrect.
Amazon Redshift Spectrum is used for querying data in S3 directly, but it does not provide automatic validation of data consistency.
- C. Correct.
Using Lambda functions to compare checksums ensures the integrity of files by verifying that the data was not altered during transfer or storage in S3.
- D. Incorrect.
SQL constraints in Redshift enforce rules during data loading, but they do not actively validate the source data for completeness or accuracy.
- E. Incorrect.
S3 Versioning provides version management for files but does not directly contribute to data validation for completeness or accuracy.