DEA-C01 Question 402
Select 3You are working as a Data Engineer for a retail company, and you are tasked with validating data in an Amazon S3 data lake before loading it into an Amazon Redshift data warehouse. The validation process must ensure that the data is complete, consistent, accurate, and maintains integrity. Which of the following steps should you implement to achieve this?
- A
Use AWS Glue DataBrew to create data quality rules for checking missing values and outliers in the dataset.
- B
Enable Amazon S3 Object Lock to ensure data cannot be modified or deleted during the validation process.
- C
Perform row count validation between the source and destination datasets to ensure data completeness.
- D
Use Amazon Redshift COPY command's
MAXERRORoption to skip invalid rows during data loading. - E
Implement checks for foreign key constraints and referential integrity in the dataset using AWS Glue ETL jobs.
Show answer and explanation
Correct answers: A, C, E
Explanation
To validate data for completeness, consistency, accuracy, and integrity, you need to use appropriate tools and techniques such as AWS Glue DataBrew for data quality rules, row count validation to confirm completeness, and checks for referential integrity to maintain data relationships. While features like Amazon S3 Object Lock and the MAXERROR option in the Redshift COPY command are useful for other purposes, they do not directly address the core requirements of data validation.
- A. Correct.
Using AWS Glue DataBrew to create data quality rules helps identify issues like missing values and outliers, ensuring data accuracy and consistency.
- B. Incorrect.
Amazon S3 Object Lock is primarily a data protection feature and does not directly help with data validation. It is not relevant for ensuring data completeness or accuracy.
- C. Correct.
Performing row count validation between the source and destination helps ensure that no records are lost during data transfer, addressing data completeness.
- D. Incorrect.
The
MAXERRORoption in the Amazon Redshift COPY command skips problematic rows but does not validate data. It is used for handling data errors, not ensuring accuracy or integrity. - E. Correct.
Implementing foreign key constraint checks and referential integrity validation ensures that relationships between datasets are intact, addressing data integrity.