DEA-C01 Question 72
Select 2You are building a data pipeline to process clickstream data collected from a web application. The pipeline ingests raw data into Amazon S3, processes it using AWS Glue, and then loads the cleaned data into Amazon Redshift for analytics. Due to data volume and processing time, you decide to introduce intermediate staging during the transformation phase. Which of the following are appropriate options for an intermediate data staging location?
- A
Amazon S3
- B
Amazon DynamoDB
- C
Amazon RDS
- D
Amazon Elastic File System (EFS)
- E
Amazon SQS
Show answer and explanation
Correct answers: A, D
Explanation
Intermediate data staging locations in a data pipeline should be capable of handling high volumes of data and integrate well with the processing services. Amazon S3 is a common choice due to its scalability, durability, and cost-effectiveness, while Amazon EFS is suitable for file-based workloads requiring scalable storage. Other services such as DynamoDB, RDS, and SQS are not suitable for intermediate staging as they are optimized for other use cases like database transactions or message queuing.
- A. Correct.
Amazon S3 is a highly durable and cost-effective storage service often used for intermediate data staging in data pipelines. It integrates seamlessly with AWS Glue and other services.
- B. Incorrect.
Amazon DynamoDB is a NoSQL database service that is not typically used for intermediate staging in a data pipeline because it is designed for low-latency key-value and document data storage, not bulk data processing or storage.
- C. Incorrect.
Amazon RDS is a relational database service best suited for transactional workloads, not for intermediate staging of large-scale data in pipelines.
- D. Correct.
Amazon Elastic File System (EFS) is a scalable file storage service that can be used for intermediate staging when applications need file-based access during processing.
- E. Incorrect.
Amazon SQS is a message queue service designed for decoupling application components, not for storing intermediate data in a data pipeline.