DEA-C01 Question 205
Select 3You are a Data Engineer tasked with setting up a data catalog for a large retail company to enable discovery and querying of data stored in Amazon S3 using Amazon Athena. The company wants the catalog to automatically update whenever new data is added to the S3 bucket. Which steps should you take to create and maintain this data catalog effectively?
- A
Use AWS Glue to create a crawler that scans the S3 bucket and populates the data catalog.
- B
Manually define each table and schema in the AWS Glue Data Catalog.
- C
Configure the AWS Glue crawler to run on a schedule or trigger it using an event-based mechanism.
- D
Ensure that the S3 bucket is set up with versioning to support the data catalog.
- E
Create an IAM role with sufficient permissions for the AWS Glue crawler to access the S3 bucket.
Show answer and explanation
Correct answers: A, C, E
Explanation
To create and maintain a data catalog for querying data in S3 with Amazon Athena, AWS Glue crawlers are the recommended solution. They automate the process of scanning data and updating the Data Catalog. Scheduling or triggering the crawler ensures the metadata stays up-to-date, and appropriate IAM permissions are necessary for the crawler to access the data in S3. Manually defining tables and enabling S3 versioning are not required steps in this process.
- A. Correct.
Correct. AWS Glue crawlers can automatically scan data in your S3 bucket and populate the Data Catalog with metadata such as table definitions and schemas.
- B. Incorrect.
Incorrect. Manually defining each table and schema is time-consuming and error-prone, especially for large datasets. AWS Glue crawlers are designed to automate this process.
- C. Correct.
Correct. Scheduling or triggering the AWS Glue crawler ensures that the data catalog is updated automatically whenever new data is added.
- D. Incorrect.
Incorrect. While S3 versioning is useful for data management, it is not a requirement for creating or maintaining the AWS Glue Data Catalog.
- E. Correct.
Correct. AWS Glue crawlers require an IAM role with the necessary permissions to access the S3 bucket and populate the Data Catalog.