Google Professional Data Engineer Question 230
Single answerGoogle Cloud PlatformYou are tasked with designing a data processing architecture for a batch ETL pipeline that processes large volumes of data once every 24 hours. The pipeline typically runs for several hours and requires a highly scalable and cost-effective solution. Which type of Dataproc cluster should you choose to optimize costs while ensuring scalability?
- A
A persistent Dataproc cluster with pre-emptible worker nodes
- B
A job-based (ephemeral) Dataproc cluster that is created and deleted automatically for each job
- C
A persistent Dataproc cluster with on-demand worker nodes
- D
A job-based (ephemeral) Dataproc cluster with manually managed lifecycle
Show answer and explanation
Correct answer: B
Explanation
For batch ETL pipelines that run infrequently (e.g., once every 24 hours), a job-based (ephemeral) Dataproc cluster is the most cost-effective and scalable choice. It ensures resources are only provisioned when needed, and Google Cloud automatically manages the creation and deletion of the cluster, reducing operational overhead. Persistent clusters, while useful for continuous or frequent processing, are not cost-efficient for sporadic workloads.
- A. Incorrect.
Using a persistent cluster with pre-emptible worker nodes can lower costs, but maintaining the cluster for 24 hours when it only processes data once a day is not cost effective.
- B. Correct.
A job-based (ephemeral) Dataproc cluster is created and deleted automatically for each job, which is ideal for batch jobs that do not require continuous uptime. This approach minimizes costs, as you only pay for the resources while the job is running.
- C. Incorrect.
While a persistent cluster with on-demand worker nodes ensures reliability, it incurs higher costs because the cluster remains active 24/7, even if it is only used once a day.
- D. Incorrect.
Manually managing the lifecycle of a job-based cluster is not ideal as it introduces operational overhead and may lead to inefficiencies if the cluster is not promptly terminated after job completion.