DEA-C01 Question 433
Single answerYou are designing a data processing pipeline for a company that processes large volumes of log data. The company wants to minimize operational overhead and focus on scaling the solution as the data grows. Which of the following AWS services would you choose, and why?
- A
Amazon EMR with Hadoop and Spark
- B
Amazon Athena
- C
Amazon EC2 with self-managed Apache Hadoop cluster
- D
Amazon Redshift
Show answer and explanation
Correct answer: B
Explanation
The company's primary requirement is to minimize operational overhead while scaling the solution. Amazon Athena is a fully serverless and managed service, making it the best choice for querying large datasets without the need to manage infrastructure. Other options, such as Amazon EMR or Amazon EC2 with Hadoop, involve varying degrees of operational management, while Amazon Redshift, though managed, is more suited for structured data warehousing needs rather than direct log data processing.
- A. Incorrect.
Amazon EMR is a managed service for big data frameworks like Hadoop and Spark, but it still requires cluster management and tuning. While it reduces some overhead compared to self-managed solutions, it doesn't minimize operational overhead as much as fully serverless services like Athena.
- B. Correct.
Amazon Athena is a fully managed and serverless service that allows you to query data directly from Amazon S3 using SQL. It requires no infrastructure management, making it ideal for minimizing operational overhead, especially for ad-hoc analytics.
- C. Incorrect.
Amazon EC2 with a self-managed Apache Hadoop cluster is an unmanaged service where you are responsible for provisioning, configuring, and maintaining the infrastructure and software. This approach has the highest operational overhead and does not align with the company's requirement to minimize operational effort.
- D. Incorrect.
Amazon Redshift is a managed data warehouse service. While it significantly reduces operational overhead compared to unmanaged solutions, it is not serverless and requires some management for scaling and optimization. It is not as lightweight as Athena for this use case.