DEA-C01 Question 382
Select 3Your team is responsible for building a data pipeline in AWS to process customer transaction data. The data is ingested into an Amazon S3 bucket in JSON format. You notice that some of the files contain missing or malformed fields. You want to ensure data quality before loading the data into an Amazon Redshift cluster for analytics. Which of the following steps would help you ensure data quality in your pipeline?
- A
Use AWS Glue DataBrew to identify and handle missing or malformed fields during preprocessing.
- B
Enable Amazon S3 Event Notifications to automatically reject files with missing or malformed fields.
- C
Implement a validation script in AWS Lambda to check the data format and structure before processing.
- D
Set up Amazon Redshift Spectrum to query the data directly in S3 and handle data quality issues during query execution.
- E
Configure AWS Glue ETL jobs to apply schema validation and cleanse the data before loading into Amazon Redshift.
Show answer and explanation
Correct answers: A, C, E
Explanation
To ensure data quality, you can use tools like AWS Glue DataBrew for preprocessing, AWS Lambda for custom validation, and AWS Glue ETL jobs to apply schema validation and cleanse the data. These steps help identify and address issues like missing or malformed fields before loading the data into Amazon Redshift. While S3 Event Notifications and Redshift Spectrum are powerful features, they are not designed for direct data quality management.
- A. Correct.
AWS Glue DataBrew is specifically designed for data preparation tasks, such as identifying and addressing missing or malformed fields in datasets, making it a suitable choice for preprocessing.
- B. Incorrect.
Amazon S3 Event Notifications can trigger actions on specific events, but they cannot automatically reject files based on their content. This option is not valid for ensuring data quality.
- C. Correct.
AWS Lambda can be used to run custom validation scripts to verify the data's format and structure before further processing, making it a valid approach for ensuring data quality.
- D. Incorrect.
Amazon Redshift Spectrum allows querying data directly in S3 but does not automatically address data quality issues. Ensuring data quality should be done before querying the data.
- E. Correct.
AWS Glue ETL jobs can be configured to apply schema validation and perform transformations, making them a valid option for cleansing data and ensuring quality before loading it into Amazon Redshift.