DEA-C01 Question 381
Select 2A company is using AWS Glue to process incoming data from multiple sources. During the ETL process, some data entries are missing critical fields, and others contain invalid data formats. As a data engineer, you need to ensure data quality before loading it into Amazon Redshift. Which of the following steps should you take to address data quality issues?
- A
Implement AWS Glue DataBrew to identify and correct missing or invalid data.
- B
Use AWS Glue Data Catalog to validate data schema against pre-defined formats.
- C
Create AWS Glue ETL jobs with transformation logic to handle missing data and validate field formats.
- D
Leverage Amazon Redshift Spectrum to directly query the data and filter out invalid records.
- E
Enable AWS Glue job bookmarks to ensure only new data is processed in subsequent runs.
Show answer and explanation
Correct answers: A, C
Explanation
Ensuring data quality involves identifying and addressing issues such as missing values or invalid formats before loading data into the target system. AWS Glue DataBrew offers a visual interface for these tasks, while AWS Glue ETL jobs allow you to implement transformation logic programmatically. The other options either do not address data quality or are focused on unrelated tasks.
- A. Correct.
Correct: AWS Glue DataBrew provides a no-code interface to visually identify, clean, and correct data quality issues such as missing values or invalid formats.
- B. Incorrect.
Incorrect: AWS Glue Data Catalog is used to store metadata and schemas, but it does not perform data validation or quality checks.
- C. Correct.
Correct: AWS Glue ETL jobs allow you to implement transformation logic, such as handling missing data or validating field formats, during the data processing stage.
- D. Incorrect.
Incorrect: Amazon Redshift Spectrum allows querying data directly from S3, but it is not designed to perform data quality checks or handle invalid records during ETL.
- E. Incorrect.
Incorrect: AWS Glue job bookmarks track processed data to avoid duplicate processing, but they do not handle data quality issues.