DEA-C01 Question 325
Single answerYou are a Data Engineer designing a real-time data processing pipeline for an e-commerce platform. The pipeline needs to handle millions of transactions per day with low latency. The platform requires you to process incoming data from a Kinesis Data Stream and store the results in Amazon S3 for further analysis. Which AWS service would you choose to process the data efficiently and why?
- A
Amazon Kinesis Data Firehose
- B
AWS Glue
- C
AWS Lambda
- D
Amazon Kinesis Data Analytics
Show answer and explanation
Correct answer: D
Explanation
Amazon Kinesis Data Analytics is the ideal choice for real-time processing of streaming data from Kinesis Data Streams. It is purpose-built for analyzing and transforming high-throughput streaming data with low latency. Unlike AWS Glue (batch ETL), Lambda (event-driven but limited for large-scale analytics), or Firehose (data delivery), Kinesis Data Analytics provides the tools needed for efficient real-time stream processing using SQL or Apache Flink.
- A. Incorrect.
Amazon Kinesis Data Firehose is primarily used for data delivery and transformation, not for real-time stream processing. It is not suitable for complex data processing tasks that require analytics or aggregations.
- B. Incorrect.
AWS Glue is a serverless data integration service typically used for ETL (Extract, Transform, Load) jobs in batch processing scenarios, not for real-time data stream processing.
- C. Incorrect.
AWS Lambda can process real-time events, but it is not ideal for large-scale analytics or aggregations on streams of data. It has limitations in terms of concurrency and execution time, which can be challenging for high-throughput, low-latency scenarios.
- D. Correct.
Amazon Kinesis Data Analytics is specifically designed for real-time analytics and processing on streaming data. It supports SQL queries and integrates seamlessly with Amazon Kinesis Data Streams for processing high-throughput data streams efficiently.