Google Professional Data Engineer Question 231
Single answerGoogle Cloud PlatformYour team is designing a data processing pipeline on Google Cloud and is evaluating the use of Dataproc clusters. The pipeline will process a large volume of data every night and requires the cluster to be available only during processing. Cost optimization is a critical requirement. Should you use a persistent or job-based Dataproc cluster, and why?
- A
Use a persistent cluster because it avoids the overhead of cluster startup time for nightly jobs.
- B
Use a persistent cluster because it allows better cost management by keeping resources reserved.
- C
Use a job-based (ephemeral) cluster because it ensures resources are available only during processing, minimizing costs.
- D
Use a job-based (ephemeral) cluster because it requires no configuration or setup for each job.
Show answer and explanation
Correct answer: C
Explanation
In this scenario, cost optimization is a key requirement, and the cluster is only needed for nightly processing. A job-based (ephemeral) cluster is the best option as it is created and billed only for the duration of the job. Persistent clusters would remain active and incur additional costs even when not in use, making them less suitable for this use case.
- A. Incorrect.
Persistent clusters avoid startup overhead, but they incur costs even when not actively processing jobs, which makes them less optimal for this scenario.
- B. Incorrect.
Persistent clusters do not help with cost optimization in this scenario, as they remain active and billable even when idle.
- C. Correct.
Job-based (ephemeral) clusters are created specifically for the duration of a job, ensuring resources are only used and billed during processing, which aligns with the cost optimization requirement.
- D. Incorrect.
While job-based clusters reduce costs, they do require some configuration or setup for each job, such as defining cluster specifications and job parameters.