DEA-C01 Question 191
Single answerA data engineering team is setting up an analytics pipeline using AWS Glue. They need to ensure that all datasets used across various ETL jobs are discoverable, described, and properly organized for reusability. Which feature of AWS Glue can best address this requirement?
- A
AWS Glue Data Catalog
- B
AWS Glue Jobs
- C
AWS Glue Crawlers
- D
AWS Glue Trigger
Show answer and explanation
Correct answer: A
Explanation
The AWS Glue Data Catalog is the core feature for centralizing metadata management and enabling dataset discovery in AWS Glue. It acts as a metadata repository for all datasets used by the ETL jobs, ensuring that datasets are properly organized, described, and reusable across different workflows.
- A. Correct.
The AWS Glue Data Catalog is specifically designed for cataloging datasets, storing metadata, and enabling data discovery. It provides a centralized metadata repository that supports dataset organization and reusability across ETL jobs.
- B. Incorrect.
AWS Glue Jobs are used to define and execute ETL tasks but do not provide metadata management or data cataloging features.
- C. Incorrect.
AWS Glue Crawlers are used to scan data sources and infer schemas, but they primarily populate the Data Catalog. They do not serve as a standalone cataloging system.
- D. Incorrect.
AWS Glue Triggers are used to schedule and manage workflows for ETL jobs but are unrelated to data cataloging.