DEA-C01 Question 66
Single answerYou are working as a data engineer in a company that processes large volumes of clickstream data collected from a web application. The data is stored in Amazon S3 in JSON format. Your team needs to process the data to calculate daily active users (DAU), filter out invalid records, and save the results back to S3 in Parquet format for downstream analytics. Which approach using Apache Spark on AWS Glue is the most appropriate?
- A
Use AWS Glue to create a crawler to catalog the data, then write an AWS Glue PySpark job to filter invalid records, compute DAU, and write the results to S3 in Parquet format.
- B
Use AWS Glue to catalog the data, then directly query the data with Amazon Athena and export the results in Parquet format to S3.
- C
Use Amazon EMR with Apache Spark to process the data and store the results in S3 in Parquet format, bypassing AWS Glue.
- D
Directly load the JSON data from S3 into Amazon Redshift, process the data with SQL queries, and export the results in Parquet format to S3.
Show answer and explanation
Correct answer: A
Explanation
The correct answer is to use AWS Glue with PySpark since it is a serverless, cost-effective solution for processing data stored in S3. AWS Glue can catalog the JSON data, perform the transformations required to filter invalid records and compute DAU using Apache Spark, and save the results in Parquet format back to S3 for analytics. The other options introduce unnecessary complexity or are not suitable for the given requirements.
- A. Correct.
This is the correct approach because AWS Glue supports PySpark for data transformation, can handle JSON data, and allows the results to be written to S3 in Parquet format. Additionally, a crawler can be used to catalog the data for schema inference.
- B. Incorrect.
This option is incorrect because Amazon Athena is a query service, not a data processing framework. While it can query data, it does not provide the flexibility to perform complex transformations required for filtering invalid records and calculating DAU.
- C. Incorrect.
This option is incorrect as using Amazon EMR introduces unnecessary overhead for this use case. AWS Glue is a more cost-effective and serverless option for Spark-based data processing in this scenario.
- D. Incorrect.
This option is incorrect because loading raw JSON data into Amazon Redshift is inefficient and not suitable for Redshift’s columnar storage optimization. Additionally, Redshift does not natively support writing data back to S3 in Parquet format.