DEA-C01 Question 215
Select 2You are a data engineer tasked with setting up a metadata and data cataloging solution for your organization's data lake on AWS. The team needs to enable search, schema discovery, and data lineage for datasets spread across Amazon S3 and AWS Glue Data Catalog. Which components should be included in your solution to meet these requirements?
- A
AWS Glue Crawlers to automatically populate the Data Catalog with metadata
- B
Custom scripts to manually update the metadata in the Data Catalog
- C
AWS Glue Data Catalog for storing and managing metadata
- D
Amazon Athena to query the data catalog and datasets directly
- E
Amazon S3 Object Tags to store metadata
Show answer and explanation
Correct answers: A, C
Explanation
To set up a metadata and data cataloging solution, AWS Glue Crawlers and AWS Glue Data Catalog are essential components. Glue Crawlers automate metadata extraction and schema discovery, while the Data Catalog stores and manages this metadata. These tools integrate seamlessly to enable search, schema discovery, and data lineage. Other options like custom scripts, Athena, or S3 Object Tags may have related use cases but are not core components of an AWS metadata and data cataloging solution.
- A. Correct.
AWS Glue Crawlers are a key component of metadata management as they automatically scan datasets, extract schema information, and populate the Glue Data Catalog with metadata. This simplifies the process of schema discovery and search.
- B. Incorrect.
While custom scripts can be used to manually update metadata, this is not efficient or scalable. AWS Glue Crawlers provide a more automated and native solution for metadata management.
- C. Correct.
AWS Glue Data Catalog is the primary service for storing and managing metadata in the AWS ecosystem. It enables schema discovery, search capabilities, and integration with other services like Athena and Redshift Spectrum.
- D. Incorrect.
Amazon Athena is a query service used to analyze data in S3 and other supported sources. While it integrates with AWS Glue Data Catalog, it is not a component for metadata management itself.
- E. Incorrect.
Amazon S3 Object Tags are not designed for managing rich metadata or supporting schema discovery. They are generally used for tagging and categorizing S3 objects.