DEA-C01 Question 84
Select 3A company wants to build a data pipeline using AWS Glue to process and transform data stored in Amazon S3. The pipeline must handle semi-structured data, track schema changes over time, and allow for querying the transformed data using Amazon Athena. Which combination of actions should the data engineer take to meet these requirements?
- A
Use AWS Glue crawlers to infer the schema from the data in Amazon S3 and catalog it in the AWS Glue Data Catalog.
- B
Write custom Apache Spark scripts in AWS Glue to handle schema changes manually.
- C
Enable schema evolution in the AWS Glue Data Catalog to automatically manage schema changes.
- D
Configure Amazon S3 Event Notifications to trigger AWS Glue jobs whenever new data is added to the S3 bucket.
- E
Query the processed data using the AWS Glue Studio interface.
Show answer and explanation
Correct answers: A, C, D
Explanation
To build a robust data pipeline using AWS Glue, the data engineer should use Glue crawlers to catalog the schema, enable schema evolution to handle changes over time, and configure S3 Event Notifications to trigger Glue jobs for processing new data. These steps ensure that the pipeline is automated, adaptable to schema changes, and integrates well with downstream querying using Amazon Athena.
- A. Correct.
Correct: AWS Glue crawlers can scan semi-structured data in Amazon S3, infer the schema, and store it in the AWS Glue Data Catalog, making it available for querying.
- B. Incorrect.
Incorrect: Writing custom scripts to handle schema changes manually is unnecessary because AWS Glue provides built-in support for schema evolution.
- C. Correct.
Correct: Enabling schema evolution in the AWS Glue Data Catalog ensures that changes to the data schema are automatically managed over time, which is essential for handling semi-structured data.
- D. Correct.
Correct: Configuring Amazon S3 Event Notifications to trigger AWS Glue jobs ensures that the pipeline processes new data as it arrives in the S3 bucket.
- E. Incorrect.
Incorrect: The AWS Glue Studio interface is used to design and visualize workflows, not for querying processed data. Amazon Athena should be used for querying the data.