MLA-C01 Question 18
Single answerYou are building a machine learning pipeline to process and train a model on a large dataset stored in an Amazon S3 bucket. The dataset consists of JSON files containing user data, but the data needs to be validated for completeness and schema conformity before training. Which approach would best ensure validated data ingestion into Amazon SageMaker?
- A
Use AWS Glue to extract, transform, and validate the data, then write the validated data back to Amazon S3 before ingesting it into SageMaker.
- B
Directly load the JSON files into Amazon SageMaker by specifying the S3 path and use SageMaker processing jobs to validate the data.
- C
Use Amazon Kinesis Data Firehose to validate the data in real-time and stream it directly into SageMaker.
- D
Manually validate the JSON files on your local machine and upload the cleaned data back to Amazon S3 for ingestion into SageMaker.
Show answer and explanation
Correct answer: A
Explanation
Using AWS Glue ensures that data is validated for completeness and schema conformity before being ingested into SageMaker. AWS Glue is designed to handle large-scale data processing and can write the cleaned and validated data back to S3, enabling a seamless and automated pipeline for machine learning workflows. This approach reduces the likelihood of errors during downstream processes and is better suited for enterprise use cases compared to the other options.
- A. Correct.
This is the correct option as AWS Glue provides capabilities to extract, transform, and validate data at scale. It can handle schema conformity and completeness checks, ensuring only validated data is stored back in Amazon S3 for further use in SageMaker.
- B. Incorrect.
While SageMaker processing jobs can handle data validation, directly loading the JSON files without pre-validation can result in errors during training. This is not the most efficient solution for large-scale datasets requiring schema validation.
- C. Incorrect.
Amazon Kinesis Data Firehose is designed for streaming data and not well-suited for batch validation of JSON files stored in S3. It is not the most appropriate service for this use case.
- D. Incorrect.
Manual validation on a local machine is not scalable for large datasets and introduces inefficiencies and the risk of human error. This approach is impractical for enterprise-level machine learning workflows.