DEA-C01 Question 380
Select 3Your organization has implemented an AWS Glue ETL job to process incoming data from multiple sources into an Amazon S3 data lake. However, you notice that some of the ingested data contains missing values and malformed records, which impacts downstream analytics. To ensure data quality, you need to implement automated checks within the ETL pipeline to identify and address these issues. Which of the following actions should you take to ensure data quality in the pipeline?
- A
Use AWS Glue's built-in FindMatches transform to detect and fix missing values in the data.
- B
Implement AWS Glue DataBrew to create data quality rules to detect anomalies, missing values, and malformed records.
- C
Integrate AWS Lambda with the ETL pipeline to validate data and log any errors for further review.
- D
Leverage AWS Glue's DynamicFrame filters to remove records that do not meet quality standards.
- E
Use Amazon Macie to classify data and identify sensitive information to ensure data quality.
Show answer and explanation
Correct answers: B, C, D
Explanation
To ensure data quality in an AWS Glue ETL pipeline, you need solutions that can detect, validate, and address issues like missing values or malformed records. AWS Glue DataBrew enables profiling and rule-based anomaly detection, AWS Lambda can add custom validation logic, and AWS Glue's DynamicFrame filters allow programmatic removal of problematic records. These tools collectively help ensure data quality at various stages of the pipeline.
- A. Incorrect.
AWS Glue's FindMatches transform is designed for record matching and deduplication, not specifically for detecting missing values or malformed records. This is an incorrect choice.
- B. Correct.
AWS Glue DataBrew provides tools for creating data quality rules, profiling data, and identifying anomalies. It is a suitable choice for ensuring data quality.
- C. Correct.
Integrating AWS Lambda with the ETL pipeline allows for custom validation logic and error logging, which can enhance data quality checks. This is a correct choice.
- D. Correct.
AWS Glue's DynamicFrame filters enable you to programmatically remove or transform records that do not meet quality standards, making it a valid approach for ensuring data quality.
- E. Incorrect.
Amazon Macie is primarily used for data discovery and classification of sensitive data, such as PII, and is not intended for ensuring data quality. This is an incorrect choice.