Symptoms & Diagnosis
Resource limit contention occurs when a container attempts to use more CPU or memory than allocated, or when the underlying node is over-provisioned. This leads to severe performance degradation or “noisy neighbor” effects where one application impacts others.
The most common symptom of CPU contention is **CPU Throttling**. Unlike memory, Kubernetes does not kill a pod for exceeding its CPU limit; it restricts the pod’s execution time. You will notice increased latency and high response times in your application logs.
Memory contention is more aggressive. If a pod exceeds its memory limit, the Linux kernel triggers an **OOMKill (Out Of Memory Kill)**. The pod will restart repeatedly, showing a status of `CrashLoopBackOff` in your cluster dashboard.
To diagnose these issues, you should check the pod’s status and events. Use the following command to see if there are any “OOMKilled” events or “Terminated” reasons.
kubectl describe pod [POD_NAME] -n [NAMESPACE]
Troubleshooting Guide
To fix resource contention, you must first identify which pods are consuming the most resources relative to their limits. Use the metrics-server to get a real-time view of resource consumption across your namespace.
kubectl top pods -n [NAMESPACE] --sort-by=cpuCompare the actual usage against the defined limits in your deployment manifest. If the usage is consistently hitting the limit, you need to adjust your resource specifications.
| Issue Type | Detection Metric | Immediate Fix |
|---|---|---|
| CPU Throttling | container_cpu_cfs_throttled_seconds_total | Increase CPU limits or optimize code concurrency. |
| Memory OOMKill | lastState: Terminated (Reason: OOMKilled) | Increase memory limits or fix memory leaks. |
| Node Pressure | Node Condition: DiskPressure / MemoryPressure | Cordon node and reschedule pods to larger nodes. |
If you find that your limits are set correctly but pods are still slow, check for **CPU Request under-provisioning**. If the sum of all pod requests on a node exceeds the node’s capacity, the Kube-scheduler has over-committed the hardware, leading to “Steal Time.”
Checking Resource YAML Configuration
Review your deployment YAML to ensure that `limits` are not set too close to `requests`. A large gap can lead to scheduling issues, while no gap can lead to immediate throttling.
resources:
requests:
memory: "256Mi"
cpu: "500m"
limits:
memory: "512Mi"
cpu: "1000m"Prevention
The best way to prevent resource contention is to implement **Vertical Pod Autoscaler (VPA)**. VPA monitors real-time usage and automatically adjusts the requests and limits for your containers, ensuring they always have the “Goldilocks” amount of resources.
Establish **Resource Quotas** at the namespace level. This prevents a single team or application from consuming all the resources of the underlying cluster, which protects other mission-critical services.
Use monitoring tools like Prometheus and Grafana to set alerts for `CPU Throttling`. Tracking the percentage of time a container is throttled allows you to proactively scale resources before the application becomes unresponsive.
Finally, always conduct load tests to determine the “peak” resource usage of your application. Setting limits based on empirical data rather than guesswork is the most effective strategy for long-term cluster stability.