DEA-C01 Question 349
Select 3You are working as a data engineer at a company that processes customer transaction data in AWS. The data is stored in an Amazon S3 bucket as Parquet files, and you need to create a report that groups transactions by customer ID, calculates the total amount spent by each customer, and computes a rolling 7-day average of transactions per customer. Which combination of services and features should you use to achieve this efficiently?
- A
Use AWS Glue to catalog the data and Amazon Athena to query it using SQL for grouping and aggregation.
- B
Use Amazon Redshift and its window functions to calculate the rolling 7-day average after loading the data.
- C
Use Amazon EMR with Apache Spark to group the data, compute the rolling 7-day average, and write the results back to Amazon S3.
- D
Use AWS Data Pipeline to orchestrate the process and manually code the logic for rolling averages in a Python script.
- E
Use Amazon QuickSight to calculate rolling averages directly by importing data from Amazon S3.
Show answer and explanation
Correct answers: A, B, C
Explanation
To efficiently process large datasets stored in Amazon S3, AWS Glue and Amazon Athena provide an easy and serverless way to perform SQL operations like grouping and aggregation. For more advanced analytics like rolling averages, Amazon Redshift offers SQL window functions, and Amazon EMR with Spark enables scalable distributed processing. AWS Data Pipeline and Amazon QuickSight are either less efficient or unsuitable for the task.
- A. Correct.
Correct: AWS Glue can catalog the Parquet files, and Amazon Athena can efficiently perform SQL-based operations like grouping and aggregation directly on the data stored in S3.
- B. Correct.
Correct: Amazon Redshift is well-suited for analytical workloads and supports SQL window functions, which are ideal for calculating rolling averages.
- C. Correct.
Correct: Amazon EMR with Apache Spark provides a scalable framework for processing large datasets using distributed computing, making it suitable for operations like grouping and rolling averages.
- D. Incorrect.
Incorrect: While AWS Data Pipeline can orchestrate workflows, manually coding the rolling average logic in a Python script is cumbersome and not an efficient use of AWS services.
- E. Incorrect.
Incorrect: Amazon QuickSight is a visualization tool and not designed for heavy data transformations such as grouping and rolling averages.