DEA-C01 Question 384
Select 3A data engineering team is using AWS Glue to process incoming data from an S3 bucket. During a recent review, stakeholders reported inconsistent and invalid data in the processed results. The team wants to ensure that only records meeting specific quality criteria (e.g., valid email formats, non-null required fields) are passed into the final dataset. Which approach should the team take to enforce data quality in the pipeline?
- A
Use AWS Glue's built-in FindMatches transform to identify and remove invalid records.
- B
Implement a custom AWS Glue PySpark job that validates data using built-in functions and drops invalid records.
- C
Utilize AWS Glue Data Quality Rules to define and enforce validation checks on the dataset.
- D
Leverage AWS Lake Formation to configure data access controls as a way to enforce data quality.
- E
Integrate AWS Glue DataBrew to create and run data profiling and quality rules on the dataset.
Show answer and explanation
Correct answers: B, C, E
Explanation
To ensure data quality in an AWS Glue pipeline, the team can use a combination of tools. AWS Glue PySpark jobs allow custom validation logic to be implemented programmatically. AWS Glue Data Quality Rules provide native support for defining and enforcing checks on the dataset. AWS Glue DataBrew is another option for profiling and cleaning data, making it suitable for preparing high-quality input data. The FindMatches transform and AWS Lake Formation are not designed for enforcing data quality directly.
- A. Incorrect.
AWS Glue's FindMatches transform is designed for identifying duplicate or matching records, not for enforcing data quality rules. It is not suitable for validating data formats or null checks.
- B. Correct.
Using a custom AWS Glue PySpark job allows you to write specific validation logic, such as checking for null values or validating email formats. This is a valid approach for enforcing data quality.
- C. Correct.
AWS Glue Data Quality Rules provide a native way to define and enforce validation checks, such as schema conformity and specific data rules. This is a suitable method for ensuring data quality.
- D. Incorrect.
AWS Lake Formation is primarily used for setting up fine-grained access controls and permissions for data. While it improves security, it does not provide mechanisms for enforcing data quality rules.
- E. Correct.
AWS Glue DataBrew is a tool for data profiling, cleaning, and quality rule enforcement. It can be used to identify and fix data quality issues before the data is processed further.