DEA-C01 Question 251
Single answerYou are designing a data lake solution on Amazon S3 to store clickstream data from a web application. The data will be processed using Amazon Athena, and new fields may be added to the data schema over time. Which approach should you use to design the schema to accommodate potential schema evolution efficiently?
- A
Use Apache Avro format for the data and define the schema in AWS Glue Data Catalog.
- B
Store the data in plain text CSV format to simplify adding new fields.
- C
Use Parquet format and allow Athena to infer the schema directly from the data.
- D
Store the data in JSON format and manually update the schema in the application code when new fields are added.
Show answer and explanation
Correct answer: A
Explanation
Apache Avro is specifically designed to handle schema evolution, supporting the addition of new fields without affecting existing data or requiring changes to the application. By defining the schema in AWS Glue Data Catalog, you centralize schema management and provide efficient querying through Athena. This approach ensures scalability and reliability as the data schema evolves over time.
- A. Correct.
Apache Avro is well-suited for schema evolution as it supports adding new fields without breaking existing applications. Storing the schema in AWS Glue Data Catalog ensures centralized schema management, which is efficient for querying via Athena.
- B. Incorrect.
While CSV is a simple format, it lacks support for schema evolution. Adding new fields would require significant manual effort to update the schema and may cause issues with backward compatibility.
- C. Incorrect.
Parquet is efficient for analytics but does not inherently support schema evolution. Allowing Athena to infer the schema directly from the data increases the risk of errors or inconsistencies when new fields are introduced.
- D. Incorrect.
JSON is flexible for schema definition but requires manual updates in the application code for every schema change, which is prone to errors and not scalable for dynamic schema evolution.