DEA-C01 Question 383
Select 2You are working as a Data Engineer for a retail company that processes large datasets of transaction logs using Amazon S3 and AWS Glue. You notice that some data records contain missing values and incorrect data types in critical fields. The company requires all datasets to meet strict data quality standards before being loaded into the data warehouse on Amazon Redshift. Which combination of actions would best ensure data quality during the ETL process?
- A
Use AWS Glue DataBrew to identify data quality issues and create transformation recipes.
- B
Configure AWS Glue to automatically reject records with missing or incorrect data types during the ETL process.
- C
Use Amazon Redshift's COPY command to validate data quality as it is being loaded into the data warehouse.
- D
Implement AWS Glue jobs to clean, standardize, and validate the data before loading it into Amazon Redshift.
- E
Enable Amazon S3 Event Notifications to trigger a Lambda function for real-time data quality checks.
Show answer and explanation
Correct answers: A, D
Explanation
Ensuring data quality during ETL processes requires tools that can identify and address data quality issues effectively. AWS Glue DataBrew is ideal for identifying quality issues and creating transformation recipes, while AWS Glue jobs can be used to implement the necessary transformations to clean and validate the data. These approaches ensure that only high-quality data is loaded into the Amazon Redshift data warehouse.
- A. Correct.
Correct: AWS Glue DataBrew is a visual data preparation tool that can identify and address data quality issues by creating transformation recipes.
- B. Incorrect.
Incorrect: AWS Glue does not natively reject records with missing or incorrect data types. You need to implement transformations or validations explicitly.
- C. Incorrect.
Incorrect: The COPY command in Amazon Redshift does not perform comprehensive data quality validation. It is primarily used for loading data efficiently.
- D. Correct.
Correct: AWS Glue jobs can be programmed to clean, standardize, and validate data, ensuring it adheres to the required quality standards before loading into Amazon Redshift.
- E. Incorrect.
Incorrect: While Lambda functions can be used for specific use cases, they are not an efficient or scalable solution for addressing data quality issues in large datasets during ETL processes.