MLS-C01 Question 44
Single answerYou are a Machine Learning Engineer tasked with building a data pipeline to preprocess large volumes of semi-structured JSON data stored in Amazon S3. The data needs to be cleaned, transformed, and prepared for training a machine learning model in Amazon SageMaker. The pipeline must handle schema evolution dynamically and ensure compatibility with downstream applications. Which feature of AWS Glue should you use to achieve this?
- A
AWS Glue Crawlers
- B
AWS Glue Data Catalog
- C
AWS Glue ETL Jobs
- D
AWS Glue ML Transforms
Show answer and explanation
Correct answer: A
Explanation
AWS Glue Crawlers are specifically designed to scan data in S3, infer its schema (including handling schema changes over time), and populate the AWS Glue Data Catalog. This ensures the data is readily available for processing by downstream applications, making it the ideal choice for this scenario.
- A. Correct.
AWS Glue Crawlers are used to automatically detect and infer the schema of data stored in S3. This feature is critical for handling schema evolution and making the data compatible with downstream applications.
- B. Incorrect.
AWS Glue Data Catalog is a metadata repository that stores schemas but does not handle schema discovery or evolution dynamically.
- C. Incorrect.
AWS Glue ETL Jobs are used for data transformation, but they require the schema to be defined beforehand and do not dynamically handle schema changes.
- D. Incorrect.
AWS Glue ML Transforms are used for advanced transformations like finding matching records but are not related to schema inference or evolution.