DEA-C01 Question 63
Select 2A company is designing a distributed data processing system on AWS to handle large-scale batch processing of log files. They want to minimize operational overhead and leverage cloud-native capabilities for scalability, fault tolerance, and cost efficiency. Which combination of services should they use to achieve this?
- A
Amazon S3 for storing log files and AWS Lambda for processing
- B
Amazon S3 for storing log files and Amazon EMR for distributed processing
- C
Amazon DynamoDB for storing log files and AWS Lambda for processing
- D
Amazon S3 for storing log files and AWS Glue for processing
- E
Amazon RDS for storing log files and Amazon EMR for processing
Show answer and explanation
Correct answers: B, D
Explanation
For large-scale batch processing, Amazon S3 is an ideal storage solution due to its scalability, durability, and cost efficiency for log files. Amazon EMR and AWS Glue are both cloud-native services designed for distributed data processing. EMR is suitable for running distributed frameworks like Hadoop and Spark, while AWS Glue provides managed ETL capabilities for processing large datasets. Combining S3 with either EMR or Glue leverages cloud-native capabilities to meet the scalability, fault tolerance, and cost efficiency requirements of the system.
- A. Incorrect.
Amazon S3 and AWS Lambda are suitable for event-driven architectures but not ideal for large-scale batch processing, as Lambda has execution time limits and is not optimized for distributed processing.
- B. Correct.
Amazon S3 and Amazon EMR are well-suited for distributed batch processing. S3 provides scalable storage for log files, and EMR is a managed service for running large-scale distributed processing frameworks like Apache Spark or Hadoop.
- C. Incorrect.
Amazon DynamoDB is a NoSQL database optimized for low-latency transactions and is not suitable for storing large files like logs. AWS Lambda also has limitations for large-scale batch processing.
- D. Correct.
Amazon S3 and AWS Glue are a good combination for this scenario. S3 provides scalable storage, and AWS Glue is a fully managed ETL service that can process large-scale datasets using distributed computing frameworks.
- E. Incorrect.
Amazon RDS is a relational database service that is not designed for storing large, unstructured log files. While EMR supports distributed processing, the choice of RDS for log storage is not appropriate.