NCA-AIIO Question 98
Select 3You are managing an AI cluster for a team of data scientists running a mix of training and inference workloads. To ensure efficient resource utilization and fairness, the cluster uses Kubernetes for orchestration and job scheduling. One of the data scientists asks why their training job was delayed, even though there are sufficient GPUs available. Which of the following factors could have caused this delay?
- A
The resource requests for the job exceeded the GPU memory available on any single node.
- B
The job was assigned a lower priority compared to other running jobs.
- C
The Kubernetes scheduler was configured to use a First-Come-First-Serve (FCFS) policy.
- D
The training job's container image was not pulled to the nodes where the job was scheduled.
- E
The cluster uses a Round-Robin scheduling strategy, which delays jobs evenly across all users.
Show answer and explanation
Correct answers: A, B, D
Explanation
Delays in job scheduling in an AI cluster orchestrated by Kubernetes can occur due to a variety of factors, including resource constraints, job priority, and node preparation (e.g., pulling container images). Understanding these factors is crucial for diagnosing and optimizing cluster operations.
- A. Correct.
Correct. If a job requests resources (e.g., GPU memory) that exceed what is available on any single node, the Kubernetes scheduler cannot place the job, leading to scheduling delays.
- B. Correct.
Correct. If the cluster uses priority-based scheduling and the job has a lower priority, it may be delayed while higher-priority jobs are scheduled first.
- C. Incorrect.
Incorrect. Kubernetes does not natively use a First-Come-First-Serve (FCFS) policy for scheduling. Instead, it uses a combination of factors like resource requests, priorities, and constraints.
- D. Correct.
Correct. If the container image for the job is not already present on the target nodes, Kubernetes will delay job execution until the image is pulled, which can take time.
- E. Incorrect.
Incorrect. Kubernetes does not use a Round-Robin scheduling strategy. Node selection is based on resource availability and constraints, not user fairness.