DEA-C01 Question 67
Single answerYou are tasked with building a data pipeline on AWS to process large amounts of semi-structured JSON data stored in Amazon S3. The data needs to be cleaned, transformed, and aggregated before being written back to another S3 bucket in a parquet format for downstream analytics. Which approach should you use to process the data using Apache Spark in this scenario?
- A
Use AWS Glue with an Apache Spark job to read the data from S3, perform transformations, and write the output back to S3 in parquet format.
- B
Use Amazon EMR with an Apache Spark application to process data directly from S3, perform transformations, and save the output back to S3.
- C
Deploy a self-managed Apache Spark cluster on Amazon EC2 instances to process the data from S3 and write the output to S3 in parquet format.
- D
Use AWS Data Pipeline to directly process the data using Apache Spark, perform transformations, and save the output in S3.
Show answer and explanation
Correct answer: A
Explanation
AWS Glue is the most efficient and cost-effective option for this scenario as it provides a fully managed, serverless Apache Spark environment tailored for big data processing. It integrates seamlessly with S3 and supports data transformations, making it ideal for your use case without the need to manage infrastructure.
- A. Correct.
This is the best approach because AWS Glue provides a serverless, fully managed platform to run Apache Spark jobs for data transformation, making it efficient and cost-effective without managing infrastructure.
- B. Incorrect.
While Amazon EMR supports Apache Spark and can process S3 data, it requires provisioning and managing the EMR cluster, which adds complexity compared to AWS Glue.
- C. Incorrect.
Although deploying a self-managed Apache Spark cluster on EC2 is possible, it requires significant operational overhead to manage the cluster, monitor performance, and handle scaling, making it less efficient than AWS Glue.
- D. Incorrect.
AWS Data Pipeline does not natively support Apache Spark jobs and is generally used for orchestrating workflows rather than directly executing Spark transformations.