MLS-C01 Question 58
Single answerA data engineering team is processing a large volume of financial transaction logs in near real-time. They need to transform the data in transit by cleaning, normalizing, and aggregating it before storing it in Amazon S3. The transformation process must scale automatically based on the incoming data volume. Which AWS service should they use to achieve this requirement?
- A
AWS Glue
- B
Amazon EMR
- C
AWS Batch
- D
Amazon Kinesis Data Analytics
Show answer and explanation
Correct answer: D
Explanation
Amazon Kinesis Data Analytics is the best choice for near real-time data transformation in transit. It allows you to process streaming data, apply transformations, and output results to destinations like Amazon S3. Other options like AWS Glue, Amazon EMR, and AWS Batch are more suited for batch processing or big data workloads and cannot meet the near real-time requirement effectively.
- A. Incorrect.
AWS Glue is typically used for batch processing and not for near real-time data transformation. It is better suited for ETL jobs involving data already stored in a data lake or database.
- B. Incorrect.
Amazon EMR is mainly used for big data processing and analytics using frameworks like Hadoop and Spark. While it can handle large-scale data, it is not inherently designed for near real-time data transformation in transit.
- C. Incorrect.
AWS Batch is designed for running batch computing jobs and is not suitable for near real-time streaming data transformation.
- D. Correct.
Amazon Kinesis Data Analytics is specifically designed for processing and transforming streaming data in near real-time. It scales automatically based on the volume of incoming data and is ideal for the described use case.