MLA-C01 Question 19
Select 2You are building a machine learning pipeline on AWS to process large-scale IoT sensor data. The data arrives continuously in JSON format from thousands of devices, and you need to clean and validate it before training your model. Which combination of AWS services should you use to efficiently ingest, validate, and store the data for downstream processing?
- A
Use Amazon Kinesis Data Streams to ingest the data, AWS Lambda to validate and clean the data, and Amazon S3 to store the processed data.
- B
Use Amazon SQS to ingest the data, Amazon EMR to validate and clean the data, and Amazon RDS to store the processed data.
- C
Use AWS Glue to ingest, validate, and clean the data, and store the processed data in Amazon Redshift.
- D
Use Amazon Kinesis Data Firehose to ingest the data, AWS Lambda to validate and clean the data, and Amazon S3 to store the processed data.
- E
Use Amazon S3 to directly ingest the data, and validate and clean the data using Amazon SageMaker Processing Jobs.
Show answer and explanation
Correct answers: A, D
Explanation
In this scenario, the goal is to handle real-time ingestion of IoT sensor data, validate and clean it, and store it for further machine learning processing. Both Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose are excellent choices for real-time ingestion, while AWS Lambda can handle lightweight validation and cleaning. Amazon S3 is a cost-effective and scalable storage solution for processed data. Other options like Amazon SQS, AWS Glue, and SageMaker Processing Jobs are either unsuitable for real-time ingestion or not optimized for this specific use case.
- A. Correct.
This is a valid solution. Amazon Kinesis Data Streams allows real-time data ingestion, AWS Lambda can process and clean the data in real-time, and Amazon S3 provides cost-effective storage for large amounts of processed data.
- B. Incorrect.
While Amazon SQS can handle message queues, it is not optimized for continuous, real-time ingestion of high-throughput data. Additionally, Amazon RDS is not suitable for storing large-scale time-series or unstructured data like JSON.
- C. Incorrect.
AWS Glue is primarily a data integration service and not optimized for real-time ingestion. It is better suited for batch ETL jobs, making it less efficient for this use case.
- D. Correct.
This is a valid solution. Amazon Kinesis Data Firehose simplifies streaming data ingestion by directly delivering it to storage destinations. AWS Lambda can validate and clean the data in real-time, and Amazon S3 is an excellent choice for storing large-scale processed data.
- E. Incorrect.
Amazon S3 does not natively support real-time data ingestion; it is primarily used for storage. While SageMaker Processing Jobs can clean and validate data, they are more suited for batch processing rather than real-time streaming.