SAP-C02 Question 101
Select 2A company is migrating its on-premises data warehouse to AWS. They plan to use Amazon Redshift for running complex analytical queries on large datasets. The data pipeline includes daily ingestion of structured data from multiple sources, and the solution must support automated schema detection and transformation before loading into Redshift. The company also wants to minimize operational overhead. Which combination of services and features should the company use to achieve this?
- A
AWS Glue for schema detection and ETL, and Amazon Redshift COPY command for loading data.
- B
Amazon EMR for schema detection and transformation, and Amazon Redshift COPY command for loading data.
- C
AWS Data Pipeline with custom scripts for schema detection and data transformation.
- D
AWS Glue Data Catalog to manage metadata, and AWS Glue jobs for ETL.
- E
Amazon Kinesis Data Firehose for direct streaming into Amazon Redshift.
Show answer and explanation
Correct answers: A, D
Explanation
The correct solution uses AWS Glue for schema detection, ETL, and metadata management through the Glue Data Catalog. Glue minimizes operational overhead by automating these processes. The Amazon Redshift COPY command is optimized for loading large datasets into Redshift efficiently. This combination aligns with the company's requirements for automation, schema detection, and minimal management effort.
- A. Correct.
Correct: AWS Glue provides automated schema detection and supports Extract, Transform, and Load (ETL) workflows. The Amazon Redshift COPY command efficiently loads bulk data into Redshift.
- B. Incorrect.
Incorrect: While Amazon EMR is a powerful tool for big data processing, it is not the most efficient or cost-effective solution for schema detection and data transformation in this scenario. It also requires more operational overhead compared to AWS Glue.
- C. Incorrect.
Incorrect: AWS Data Pipeline can handle ETL, but it requires custom scripting and significant management effort, which increases operational overhead.
- D. Correct.
Correct: AWS Glue Data Catalog centralizes metadata management for Redshift and other data sources, and Glue jobs can automate the ETL processes, minimizing operational overhead.
- E. Incorrect.
Incorrect: Amazon Kinesis Data Firehose is designed for near-real-time streaming, which is not a requirement in this case. It also does not support advanced schema detection or transformation.