DEA-C01 Question 252
Select 2You are designing a data lake architecture on AWS to store and analyze customer transaction data. The data is stored in Amazon S3 in Parquet format and queried using Amazon Athena. Over time, the schema of the data may evolve as new fields are added or existing fields are modified. How can you design the data model to handle schema evolution efficiently while minimizing query failures?
- A
Use AWS Glue Data Catalog to manage the schema and enable versioning for schema updates.
- B
Store all historical schema versions in a separate S3 bucket to ensure backward compatibility.
- C
Enable schema merging in Athena queries for handling schema changes dynamically.
- D
Adopt a schema-on-read approach to allow flexibility in schema evolution.
- E
Convert the data format from Parquet to JSON to handle schema evolution more easily.
Show answer and explanation
Correct answers: A, D
Explanation
To handle schema evolution efficiently in a data lake on AWS, you can leverage AWS Glue Data Catalog to manage schema versions and adopt a schema-on-read approach. These practices ensure flexibility in querying evolving data structures while minimizing the risk of query failures caused by schema changes.
- A. Correct.
Using AWS Glue Data Catalog with schema versioning helps manage schema changes over time and ensures compatibility with Athena queries.
- B. Incorrect.
Storing historical schema versions in a separate S3 bucket is not a standard or efficient approach for schema evolution in a data lake. AWS Glue Data Catalog provides better support for this purpose.
- C. Incorrect.
Athena does not have a built-in feature for schema merging. Schema evolution must be managed through other tools like AWS Glue or by following best practices for schema-on-read.
- D. Correct.
Adopting a schema-on-read approach ensures that queries are flexible and can handle schema evolution without requiring strict adherence to a predefined schema.
- E. Incorrect.
Converting Parquet to JSON is unnecessary and inefficient. Parquet is designed to handle schema evolution effectively, and schema-on-read can be used to address changes.