DEA-C01 Question 291
Select 2A company processes large volumes of log data generated by its applications and wants to automate the ingestion and transformation of this data into a queryable format. The company needs to handle data in near real-time and store it in a data warehouse for analytics. Which combination of AWS services should the company use to automate this data processing pipeline?
- A
Amazon Kinesis Data Streams to ingest data, AWS Lambda for transformation, and Amazon Redshift for storage
- B
AWS Glue to ingest data, AWS Step Functions for orchestration, and Amazon S3 for storage
- C
Amazon Kinesis Firehose to ingest data, AWS Glue for transformation, and Amazon S3 for storage
- D
Amazon Kinesis Firehose to ingest data, AWS Lambda for transformation, and Amazon Redshift for storage
- E
Amazon SQS to ingest data, AWS Lambda for transformation, and Amazon DynamoDB for storage
Show answer and explanation
Correct answers: A, D
Explanation
To automate a near real-time data processing pipeline, services like Amazon Kinesis Data Streams or Amazon Kinesis Firehose are ideal for ingesting streaming data. AWS Lambda can transform data on the fly, and Amazon Redshift is a suitable destination for analytics due to its data warehousing capabilities. This combination ensures scalability and aligns with the requirements of real-time processing and storage for analytical querying.
- A. Correct.
Correct: Amazon Kinesis Data Streams can handle real-time data ingestion, AWS Lambda can process and transform the data, and Amazon Redshift is optimized for storing and querying structured analytics data.
- B. Incorrect.
Incorrect: AWS Glue is designed for ETL and batch processing, not real-time ingestion. While Step Functions can help orchestrate workflows, this solution does not align with the near real-time requirements.
- C. Incorrect.
Incorrect: Although Amazon Kinesis Firehose and AWS Glue are valid services, AWS Glue is better suited for batch processing, not real-time transformations. The data ultimately stored in Amazon S3 may not be optimal for analytics compared to Amazon Redshift.
- D. Correct.
Correct: Amazon Kinesis Firehose is managed and supports near real-time ingestion, AWS Lambda can handle data transformation, and Amazon Redshift is ideal for analytics use cases.
- E. Incorrect.
Incorrect: Amazon SQS is a message queue service and not suitable for real-time streaming. Additionally, Amazon DynamoDB is a NoSQL database and not a data warehouse optimized for analytics.