DEA-C01 Question 282
Select 2You are working on an AWS Glue ETL pipeline that processes data stored in Amazon S3 using an Apache Parquet schema. Over time, the data schema has evolved with the addition of new columns. You want to ensure the pipeline continues to process both old and new data without errors. Which schema evolution techniques should you use?
- A
Enable AWS Glue's 'schema compatibility' feature during crawler configuration.
- B
Use the 'MergeSchema' option provided by Apache Spark during the Glue ETL job.
- C
Manually update the schema definition in the AWS Glue Data Catalog for each change.
- D
Store the schema in AWS Lake Formation and configure schema enforcement policies.
- E
Configure the Parquet output format to append new columns while maintaining backward compatibility.
Show answer and explanation
Correct answers: B, E
Explanation
Handling schema evolution effectively in AWS Glue requires techniques that allow the ETL pipeline to process both old and new schema versions without manual intervention. The 'MergeSchema' option in Apache Spark helps Glue ETL jobs dynamically handle schema changes, while configuring the Parquet output format to maintain backward compatibility ensures the data remains usable across schema versions. These approaches eliminate the need for manual updates and are scalable for large datasets.
- A. Incorrect.
AWS Glue does not have a specific 'schema compatibility' feature; schema evolution is managed by tools like Apache Spark or by manually updating the Data Catalog.
- B. Correct.
The 'MergeSchema' option in Apache Spark allows the Glue ETL job to handle schema evolution by merging schemas from different data files, making it a valid approach.
- C. Incorrect.
Manually updating the schema in the Glue Data Catalog is error-prone and not scalable for frequent schema changes, making it an inefficient approach.
- D. Incorrect.
While AWS Lake Formation can enforce schema policies, it does not directly support schema evolution techniques for ETL jobs using Parquet data.
- E. Correct.
Configuring the Parquet output format to append new columns ensures backward compatibility, making it a suitable technique for handling schema evolution.