DEA-C01 Question 404
Select 2You are a data engineer working on a pipeline that ingests customer transaction data into an Amazon S3 bucket. The data is then processed using AWS Glue and stored in Amazon Redshift for analytics. During a recent audit, your team discovered that some transaction records have missing fields and inconsistent formatting (e.g., dates in different formats). To ensure data completeness, consistency, and integrity, which steps should you take in your pipeline?
- A
Implement AWS Glue DataBrew to profile and clean the data before processing.
- B
Enable Amazon S3 Transfer Acceleration to speed up data ingestion.
- C
Use AWS Glue's built-in data quality transformations to validate and correct records during the ETL process.
- D
Configure Amazon Redshift's COPY command to reject records with missing or invalid fields.
- E
Build a Lambda function to monitor and retry failed S3 file uploads.
Show answer and explanation
Correct answers: A, C
Explanation
To ensure data completeness, consistency, and integrity in the pipeline, you should focus on both profiling and cleaning the data before processing and validating it during the ETL process. AWS Glue DataBrew allows you to detect and fix issues like missing fields and inconsistent formatting proactively. Additionally, AWS Glue's built-in data quality transformations help enforce these rules during the ETL stage, ensuring that only cleaned and validated data is passed downstream. While other options may improve performance or error handling, they do not directly address the identified data quality issues.
- A. Correct.
AWS Glue DataBrew can be used to profile, clean, and standardize data, ensuring completeness and consistency before processing. This addresses issues like missing fields and inconsistent formatting.
- B. Incorrect.
Amazon S3 Transfer Acceleration speeds up data transfers but does not address data validation or consistency issues. This is not relevant to solving the problem.
- C. Correct.
AWS Glue provides built-in data quality transformations that can validate, clean, and enforce consistency during the ETL process. This is critical for ensuring data integrity.
- D. Incorrect.
Amazon Redshift's COPY command can reject invalid records, but this approach does not proactively clean or validate data before it reaches Redshift. It only prevents bad data from being loaded, which does not fully solve the problem.
- E. Incorrect.
Building a Lambda function to monitor S3 uploads is useful for error handling in file transfers but does not address data completeness, consistency, or formatting issues.