DEA-C01 Question 400
Select 3You are a data engineer working for an e-commerce company. The company ingests sales data from multiple sources into an Amazon S3 bucket. You need to ensure the data is complete, consistent, accurate, and adheres to integrity rules before it is processed by a downstream ETL pipeline using AWS Glue. Which of the following is the MOST appropriate set of actions you should take to validate the data?
- A
Use AWS Glue DataBrew to identify missing values, inconsistent formatting, and duplicate records.
- B
Leverage Amazon S3 Event Notifications to automatically validate data completeness on object upload.
- C
Implement AWS Glue ETL scripts to compare incoming data with a pre-defined schema for validation.
- D
Use Amazon Athena to query the data and create validation rules to identify anomalies.
- E
Enable S3 Object Lock to ensure data integrity during the validation process.
Show answer and explanation
Correct answers: A, C, D
Explanation
To ensure data completeness, consistency, accuracy, and integrity before processing, you should use tools like AWS Glue DataBrew for profiling, AWS Glue ETL scripts for schema validation, and Amazon Athena for querying and anomaly detection. These tools provide complementary capabilities to identify and resolve data quality issues. While S3 Event Notifications and Object Lock serve other purposes, they do not directly validate the quality of the data.
- A. Correct.
Correct. AWS Glue DataBrew is a data preparation tool that can help identify issues like missing values, inconsistent formatting, and duplicates, which are critical for ensuring data completeness and consistency.
- B. Incorrect.
Incorrect. While S3 Event Notifications can trigger actions upon object upload, they are not designed for validating data completeness directly. You would still need additional tools or logic for validation.
- C. Correct.
Correct. AWS Glue ETL scripts can compare incoming data against a schema to validate structure, accuracy, and consistency before processing.
- D. Correct.
Correct. Amazon Athena allows querying data directly in S3 to create validation rules and identify anomalies, which is a powerful method for ensuring data quality.
- E. Incorrect.
Incorrect. S3 Object Lock ensures that data cannot be deleted or overwritten, which is useful for compliance but not directly related to validation of completeness, accuracy, or consistency.