DEA-C01 Question 115
Single answerYou are working as a data engineer for a company that processes large-scale clickstream data. The data is streamed in real-time from multiple sources into an Amazon Kinesis Data Stream. You need to store this data in Amazon S3 for long-term analysis while ensuring the data is partitioned by event date for efficient querying. Which solution provides the most efficient and cost-effective way to achieve this?
- A
Use an AWS Lambda function to process data from Kinesis Data Streams and write it to Amazon S3 with the event date as the prefix.
- B
Use Amazon Kinesis Data Firehose to directly deliver the data to Amazon S3 with dynamic partitioning enabled based on the event date.
- C
Stream the data from Kinesis Data Streams into Amazon Redshift, then export the data to Amazon S3 with event date partitioning.
- D
Set up an Amazon EMR cluster to consume the data from Kinesis Data Streams and write it to Amazon S3 partitioned by event date.
Show answer and explanation
Correct answer: B
Explanation
Amazon Kinesis Data Firehose is the most efficient and cost-effective service for delivering streaming data to Amazon S3. It supports dynamic partitioning based on attributes, such as the event date, without requiring additional infrastructure or manual coding. This makes it the best choice for the given scenario.
- A. Incorrect.
This solution is feasible, but it requires you to manually write and maintain the Lambda function code for partitioning the data by event date. Additionally, Lambda may not scale efficiently with very high data volumes compared to other AWS services.
- B. Correct.
This is the correct solution. Amazon Kinesis Data Firehose natively supports delivering data to Amazon S3 and can dynamically partition the data based on event attributes such as the event date, making it both efficient and cost-effective.
- C. Incorrect.
While Amazon Redshift can consume data from Kinesis Data Streams, it is not designed for long-term storage or efficient partitioning in S3. This approach adds unnecessary complexity and cost.
- D. Incorrect.
Using Amazon EMR for this task introduces additional overhead in terms of cost and configuration. While it can achieve the desired partitioning, it is not the most efficient solution for this use case.