DEA-C01 Question 401
Select 2You are working as a data engineer for an e-commerce company that collects transaction data from multiple sources into an Amazon S3 bucket. The data is then processed using AWS Glue and stored in an Amazon Redshift data warehouse. During one of the ETL jobs, you notice that some rows in the dataset are missing critical fields, such as 'customer_id' or 'order_id'. You want to ensure that future ETL jobs validate the data for completeness, consistency, accuracy, and integrity before loading it into the data warehouse. Which of the following actions should you take to address this issue?
- A
Use AWS Glue's built-in data quality feature to define rules for required fields and reject rows with missing critical fields.
- B
Implement an AWS Lambda function to monitor the S3 bucket for new data files and validate the schema before triggering the Glue job.
- C
Enable Amazon Redshift's automatic data validation feature to reject invalid rows during the COPY operation.
- D
Incorporate data validation logic directly into the AWS Glue ETL script to check for missing fields and log issues.
- E
Use Amazon QuickSight to create a dashboard that monitors data quality metrics after the data is loaded into Redshift.
Show answer and explanation
Correct answers: A, D
Explanation
To ensure data completeness, consistency, accuracy, and integrity during ETL jobs, AWS Glue's built-in data quality features and custom data validation logic in the ETL script are the most appropriate solutions. These options allow you to proactively identify and handle missing or inconsistent data before it reaches the data warehouse. Monitoring tools like Amazon QuickSight are better suited for analyzing data quality after the data has been processed, which is not a preventive measure.
- A. Correct.
AWS Glue's built-in data quality feature allows you to define rules to validate data during the ETL process, ensuring completeness and integrity. This is an effective way to reject rows with missing critical fields.
- B. Incorrect.
Using an AWS Lambda function to validate the schema before triggering the Glue job is not the most efficient or scalable approach for validating data completeness and consistency during ETL jobs.
- C. Incorrect.
Amazon Redshift does not have an automatic data validation feature during the COPY operation that specifically rejects rows based on missing fields. Validation must be handled before loading data into Redshift.
- D. Correct.
Incorporating data validation logic in the AWS Glue ETL script allows you to programmatically check for missing fields and address any data quality issues directly during the transformation process.
- E. Incorrect.
Amazon QuickSight is primarily a business intelligence tool for creating dashboards and visualizing data. While it can monitor data quality metrics, it does not perform validation before loading the data into Redshift.