MLS-C01 Question 170
Single answerA data engineering team at a financial services company is building a recommendation engine that processes large-scale customer transaction data in real-time. The team needs to preprocess the data, perform complex aggregations, and handle distributed data processing efficiently. After preprocessing, the data will be passed to a machine learning model hosted on an endpoint for predictions. Which compute platform should they choose for the preprocessing step?
- A
Amazon EMR with Apache Spark
- B
AWS Lambda with Python
- C
Amazon SageMaker Processing with built-in container
- D
Amazon EC2 instances with custom scripts
Show answer and explanation
Correct answer: A
Explanation
Amazon EMR with Apache Spark is the most appropriate choice for preprocessing large-scale, real-time data with complex aggregations. Apache Spark, running on Amazon EMR, is specifically designed for distributed computing and can efficiently process large volumes of data in parallel. Other options, such as AWS Lambda or EC2 instances with custom scripts, either lack the scalability and efficiency needed for distributed processing or require more manual effort to configure and manage.
- A. Correct.
Amazon EMR with Apache Spark is the best choice for distributed data processing and handling large-scale real-time data with complex aggregations. Spark's capabilities for distributed computing make it highly efficient for this use case.
- B. Incorrect.
AWS Lambda is not ideal for this scenario because it is designed for lightweight, serverless functions with limited execution time (15 minutes max) and is unsuitable for large-scale distributed processing.
- C. Incorrect.
Amazon SageMaker Processing is a good option for preprocessing data, but it is not optimized for handling real-time, large-scale distributed data. It is better suited for batch processing tasks.
- D. Incorrect.
Amazon EC2 instances with custom scripts can handle distributed data processing but require significant manual effort to set up and manage distributed computing frameworks, unlike Spark on EMR, which provides built-in capabilities.