Pod stuck Pending on a cluster with idle nodes: the requests nobody added up
Problem
A deployment created 3 replicas and one pod sat in Pending forever:
FailedScheduling: 0/12 nodes are available: 4 Insufficient cpu, 3 Insufficient memory,
5 node(s) had untolerated taint. preemption: 0/12 nodes are available.Node monitoring showed 40% CPU cluster-wide. The scheduler disagreed, and the scheduler had better data.
Root cause
Scheduling uses requests, not measured usage. Requests are reservations the kubelet enforces, and they silently accumulate: a pod requesting 2 CPU but using 0.1 still occupies 2 CPU of schedulable capacity. Our namespace had grown dozens of over-requested deployments (requests set high "to be safe", never revisited), so the cluster's allocatable capacity was 85% committed while its measured utilization was 40%. The taints clause is the other half — those 5 nodes are system/spot pools that ordinary workloads can never land on, and people keep counting them mentally as available.
kubectl describe nodes | grep -A5 "Allocated resources"
kubectl top nodes
… 1 more line in the fix🔒 the fix — including 3 code blocks — is members-only. $1/mo unlocks everything.