MLS-C01 Question 172
Single answerYou are tasked with building a machine learning pipeline to preprocess and analyze a large dataset consisting of terabytes of semi-structured log data generated by IoT devices. The preprocessing involves distributed data transformations, and the model training requires iterative computations. Which compute platform should you choose to effectively handle this workload?
- A
Amazon SageMaker with built-in XGBoost algorithm
- B
AWS Glue for ETL and Amazon EMR with Apache Spark
- C
AWS Lambda with custom Python scripts
- D
Amazon EC2 instances running TensorFlow
Show answer and explanation
Correct answer: B
Explanation
The scenario involves preprocessing terabytes of semi-structured data and performing iterative computations, which requires a distributed computing framework. AWS Glue can handle ETL tasks for large datasets, and Apache Spark on Amazon EMR is specifically designed for distributed data processing and iterative machine learning workloads. Combining these services provides an optimal solution for the given requirements.
- A. Incorrect.
Amazon SageMaker with built-in XGBoost is optimized for tabular data and machine learning tasks but does not inherently support distributed data processing or iterative computations directly on terabytes of semi-structured data.
- B. Correct.
AWS Glue is designed for ETL (Extract, Transform, Load) tasks on large datasets, and Amazon EMR with Apache Spark provides a distributed computing framework ideal for large-scale data preprocessing and iterative ML tasks. This is the most effective choice for the described workload.
- C. Incorrect.
AWS Lambda is designed for lightweight, event-driven functions and is not suitable for large-scale, distributed data processing or iterative machine learning computations due to its execution time and resource limits.
- D. Incorrect.
Amazon EC2 instances running TensorFlow can handle model training but do not provide a distributed framework for preprocessing or iterative computations on terabytes of data, making it a suboptimal choice for this scenario.