DEA-C01 Question 290
Select 2A data engineering team is responsible for managing a large-scale data pipeline that ingests and transforms data daily. The team has noticed that some transformations are failing intermittently due to schema mismatches in the source data. To address this, the team wants to implement a solution that can validate the schema of incoming data before ingestion and isolate invalid records for further analysis. Which AWS service or combination of features should the team use to achieve this?
- A
AWS Glue Schema Registry to enforce schema validation during data processing
- B
Amazon Kinesis Data Firehose with data transformation enabled to validate and isolate invalid records
- C
Amazon S3 Event Notifications to trigger a Lambda function for schema validation
- D
AWS Lake Formation to enforce schema validation at the table level
- E
AWS Databrew to perform schema validation and clean the incoming data
Show answer and explanation
Correct answers: A, B
Explanation
To handle schema validation and isolate invalid records, AWS Glue Schema Registry can enforce schema compliance during data processing, while Amazon Kinesis Data Firehose offers the ability to validate, transform, and redirect invalid data during ingestion. These services are purpose-built for such use cases and integrate seamlessly into data pipelines. Other options like S3 Event Notifications and AWS Databrew require more custom implementations or are not directly suited for schema validation in real-time ingestion scenarios.
- A. Correct.
AWS Glue Schema Registry can enforce schema validation during data processing, ensuring that incoming data conforms to an expected schema. This helps in identifying schema mismatches earlier in the pipeline.
- B. Correct.
Amazon Kinesis Data Firehose provides a built-in data transformation feature that can validate and transform incoming data before delivering it to the destination. Invalid records can be redirected to an S3 bucket for further analysis.
- C. Incorrect.
Amazon S3 Event Notifications can trigger a Lambda function, but this solution requires custom implementation for schema validation and is not as streamlined as other services like AWS Glue Schema Registry and Kinesis Data Firehose.
- D. Incorrect.
AWS Lake Formation focuses on access control and data governance but does not natively enforce schema validation during data ingestion.
- E. Incorrect.
AWS Databrew is a data preparation tool for cleaning and transforming data, but it is not specifically designed for real-time schema validation during ingestion.