DEA-C01 Question 270
Select 2Your organization processes large volumes of transactional data in an AWS Glue ETL pipeline and stores the results in an Amazon S3 data lake. The analytics team has raised concerns about data accuracy and trustworthiness, especially when troubleshooting issues or validating reports. Which actions can you take to ensure data lineage is established and helps address these concerns?
- A
Enable AWS Glue Data Catalog table versioning to track schema changes over time.
- B
Use AWS Glue job bookmarks to maintain state between job runs and track processing progress.
- C
Integrate AWS Glue with Amazon Athena to allow querying the data directly for validation purposes.
- D
Configure AWS Glue to write audit logs to Amazon CloudWatch and enable detailed logging for job execution.
- E
Implement AWS Glue DataBrew to visually inspect and trace data transformations in the pipeline.
Show answer and explanation
Correct answers: A, E
Explanation
To ensure data accuracy and trustworthiness, understanding and tracking data lineage is critical. Enabling AWS Glue Data Catalog table versioning helps track schema changes over time, providing insights into how the data has evolved. Similarly, AWS Glue DataBrew provides a powerful visual tool to trace and inspect data transformations, further ensuring data accuracy and lineage. Other options, such as job bookmarks, Athena integration, and CloudWatch logs, may support the pipeline in other ways but do not directly address data lineage concerns.
- A. Correct.
Enabling AWS Glue Data Catalog table versioning allows you to track schema changes over time, which is critical for understanding data lineage and ensuring the trustworthiness of the data.
- B. Incorrect.
AWS Glue job bookmarks are useful for incremental data processing but do not directly provide data lineage or insights into data accuracy.
- C. Incorrect.
While Amazon Athena can be used to query and validate data, it does not inherently provide data lineage or traceability of transformations.
- D. Incorrect.
Audit logs in Amazon CloudWatch can help monitor job execution but do not provide detailed data lineage or visibility into data transformations.
- E. Correct.
AWS Glue DataBrew provides a visual interface for inspecting and tracing data transformations, which is valuable for establishing data lineage and ensuring accuracy.