DEA-C01 Question 223
Select 3You are a Data Engineer at a company that collects large-scale IoT sensor data from devices distributed across multiple regions. The data is ingested into an Amazon Kinesis Data Stream, and you need to build a scalable and fault-tolerant data pipeline to process the data for real-time analytics. The processed data should then be stored in Amazon S3 for long-term storage. Which combination of services and approaches would best meet these requirements?
- A
Use AWS Lambda to process records from the Kinesis Data Stream and write the processed data to Amazon S3.
- B
Use Amazon Kinesis Data Analytics to process the data in real-time and output the results to an Amazon Kinesis Data Firehose.
- C
Configure Amazon Kinesis Data Firehose to write directly from the Kinesis Data Stream to Amazon S3, bypassing the need for additional processing.
- D
Deploy an Amazon EMR cluster to read data from the Kinesis Data Stream, process it using Apache Spark, and store the results in Amazon S3.
- E
Use Amazon Kinesis Data Firehose to transform the data using its built-in data transformation feature and deliver the transformed data to Amazon S3.
Show answer and explanation
Correct answers: A, B, D
Explanation
To build a scalable and fault-tolerant data pipeline for real-time analytics, you can use AWS Lambda as an event-driven compute service to process records from the Kinesis Data Stream and store the processed results in Amazon S3. Alternatively, Amazon Kinesis Data Analytics can process the data in real time using SQL and output the results to a Kinesis Data Firehose for delivery to Amazon S3. For more complex processing needs, Amazon EMR with Apache Spark can be used to process the data and write to Amazon S3. Each solution depends on the specific processing requirements and complexity, but all three options are valid for constructing the pipeline.
- A. Correct.
Correct: AWS Lambda can read from a Kinesis Data Stream, process the data in real-time, and write the results to Amazon S3. This approach is scalable and serverless, making it a good option for real-time data pipelines.
- B. Correct.
Correct: Amazon Kinesis Data Analytics provides a managed service for real-time stream processing using SQL. It can process data in real-time from a Kinesis Data Stream and output results to other destinations, such as Kinesis Data Firehose.
- C. Incorrect.
Incorrect: Kinesis Data Firehose cannot directly read from a Kinesis Data Stream. It is designed to ingest data directly from producers or other services like Kinesis Data Analytics.
- D. Correct.
Correct: Amazon EMR with Apache Spark can be used for batch and real-time stream processing. It is suitable for complex processing tasks and integrates well with Kinesis Data Streams and Amazon S3.
- E. Incorrect.
Incorrect: While Kinesis Data Firehose has a built-in data transformation feature, it is not designed to pull data directly from a Kinesis Data Stream, which makes this approach unsuitable for the described architecture.