DEA-C01 Question 42
Single answerYou are a data engineer at a healthcare company that processes large amounts of patient data. The data is stored in an Amazon S3 bucket in JSON format. You need to transform this data by flattening the nested JSON structure and converting it into a Parquet file for downstream analytics. Which AWS service and feature combination is the most efficient for this task?
- A
Use AWS Glue with a custom ETL script to process the JSON data and write the output to Parquet format in Amazon S3.
- B
Use Amazon Athena to query the JSON data and export the query results directly as Parquet files.
- C
Use Amazon Kinesis Data Analytics to process the JSON data in real-time and write the transformed data to Parquet format.
- D
Use AWS Lambda to read the JSON files from S3, transform them using Python code, and save the results back to S3 in Parquet format.
Show answer and explanation
Correct answer: A
Explanation
AWS Glue is the most suitable service for transforming and processing data stored in Amazon S3, especially when flattening nested JSON structures and converting them to a columnar format like Parquet. AWS Glue provides managed ETL capabilities and is optimized for such workloads. Other options, like Athena or Lambda, are either not designed for large-scale transformations or have inherent limitations (e.g., real-time focus or resource constraints).
- A. Correct.
This is the correct answer. AWS Glue is a managed ETL service that allows you to create and execute ETL jobs. It is ideal for transforming data stored in Amazon S3, particularly for use cases like converting JSON to Parquet.
- B. Incorrect.
While Amazon Athena supports querying JSON data and can produce results in Parquet format, it is not designed for large-scale data transformation or flattening nested structures efficiently.
- C. Incorrect.
Amazon Kinesis Data Analytics is designed for real-time data processing, not batch transformation of data stored in Amazon S3.
- D. Incorrect.
AWS Lambda could technically be used for this task, but it is not efficient for processing large datasets due to its execution time limits and memory constraints. It is better suited for event-driven tasks.