DEA-C01 Question 61
Select 2You are designing a data processing pipeline for an e-commerce company that needs to analyze terabytes of clickstream data from users in real-time. The architecture must handle high availability, scalability, and fault tolerance while ensuring minimal operational overhead. Which AWS services and design choices would best suit this requirement?
- A
Use Amazon Kinesis Data Streams to ingest and process clickstream data in real-time.
- B
Deploy Apache Kafka on Amazon EC2 for real-time data ingestion and processing.
- C
Use Amazon S3 to store raw clickstream data for batch processing instead of real-time processing.
- D
Leverage AWS Lambda to process data in real-time from the ingestion stream.
- E
Use Auto Scaling for EC2 instances running a custom data processing application.
Show answer and explanation
Correct answers: A, D
Explanation
To meet the requirements of real-time data processing with high availability, scalability, and minimal operational overhead, using Amazon Kinesis Data Streams for ingestion and AWS Lambda for real-time processing is the best fit. These services are fully managed, scalable, and fault-tolerant, ensuring the pipeline can handle terabytes of data with reduced operational complexity. Other options like deploying Apache Kafka on EC2 or using Auto Scaling for EC2 introduce more operational overhead, while Amazon S3 is suited for batch processing instead of real-time processing.
- A. Correct.
Amazon Kinesis Data Streams is an ideal service for ingesting and processing real-time data. It is fully managed, highly scalable, and fault-tolerant, making it a strong choice for this use case.
- B. Incorrect.
While Apache Kafka is a popular tool for real-time data processing, deploying it on Amazon EC2 requires significant operational overhead for setup, maintenance, and scaling, making it less ideal compared to managed services like Kinesis.
- C. Incorrect.
Amazon S3 is excellent for storing raw data for batch processing but does not support real-time processing capabilities. This option does not meet the requirement for real-time data analysis.
- D. Correct.
AWS Lambda can process data in real-time when triggered by events from services like Kinesis Data Streams. It is serverless, scalable, and reduces operational overhead, aligning well with the requirements.
- E. Incorrect.
Auto Scaling for EC2 instances can enable scalability, but managing a custom data processing application on EC2 requires significant operational effort and does not inherently provide real-time processing capabilities.