Google Professional Data Engineer Question 229
Single answerGoogle Cloud PlatformYou are designing a data processing pipeline for your company's batch and ad-hoc analytics workloads using Google Cloud Dataproc. The batch jobs are scheduled to run every night and can take several hours, while ad-hoc jobs are submitted sporadically during working hours. Cost efficiency and operational simplicity are top priorities. How should you configure your Dataproc clusters?
- A
Use a single persistent Dataproc cluster for both batch and ad-hoc workloads to avoid cluster provisioning delays.
- B
Use a persistent Dataproc cluster for batch workloads and create ephemeral (job-based) clusters for ad-hoc workloads.
- C
Use ephemeral (job-based) clusters for both batch and ad-hoc workloads.
- D
Use two separate persistent Dataproc clusters: one for batch workloads and another for ad-hoc workloads.
Show answer and explanation
Correct answer: B
Explanation
The best solution balances cost efficiency and operational simplicity. A persistent cluster is ideal for predictable, scheduled batch jobs to avoid provisioning delays, while ephemeral clusters for ad-hoc workloads ensure resources are only provisioned when needed, reducing idle costs.
- A. Incorrect.
Using a single persistent Dataproc cluster for both workloads simplifies management but may lead to underutilization during idle periods, increasing costs. It also risks resource contention between batch and ad-hoc jobs.
- B. Correct.
Using a persistent cluster for batch workloads ensures availability for scheduled jobs, while ephemeral clusters for ad-hoc jobs minimize idle costs and provide flexibility by provisioning only when needed.
- C. Incorrect.
Using ephemeral clusters for both workloads eliminates idle costs but could introduce delays for batch jobs due to cluster provisioning time, which is less suitable for regular, time-sensitive schedules.
- D. Incorrect.
Using two separate persistent clusters eliminates resource contention but significantly increases costs due to the always-on nature of both clusters, even when idle.