Few things are more stressful in a production environment than a pod that keeps crashing. The logs are incomplete, the application is down, and every minute costs you.
The good news is that Kubernetes gives you a surprisingly rich set of tools to diagnose exactly what went wrong — if you know where to look and what to look for. This guide walks you through a systematic, battle-tested approach to debugging crashing pods, from the first alert to the root cause.
Why Pods Crash: The Big Picture
Before diving into commands and workflows, it helps to understand the categories of failure that cause pods to crash. Most crashing pods fall into one of five buckets: application errors, misconfigured environment variables or missing secrets, insufficient resources, bad container images, and dependency failures where the app starts but a required service is unreachable.
Understanding which category you are dealing with narrows your debugging path dramatically. A pod stuck in CrashLoopBackOff because of a missing environment variable requires a completely different fix than one being OOMKilled because it ran out of memory.
The first step is always to gather enough information to identify which bucket your failure belongs to. It is also worth noting that a misconfigured CI/CD pipeline is one of the most common sources of bad deployments that lead to crashing pods. If your pipelines are not well-structured, they become a recurring source of pod failures — understanding CI/CD pipeline best practices is just as important as knowing how to debug the fallout.
Understanding Pod Status and What It Tells You
The very first thing you should do when a pod is misbehaving is check its status. Running kubectl get pods in the relevant namespace gives you an immediate snapshot of what is happening. Kubernetes surfaces a lot of diagnostic information in just this one command if you know how to read it.
A status of CrashLoopBackOff means the container started, crashed, and Kubernetes kept trying to restart it. The backoff is Kubernetes being kind — it waits progressively longer between restarts to avoid hammering a broken service. This status almost always means the application itself is failing on startup.
A status of OOMKilled means the container exceeded its memory limit and was forcibly killed by the kernel. This is a resource configuration problem, not necessarily an application bug — though it can be both.
A status of ImagePullBackOff or ErrImagePull means Kubernetes cannot pull the container image. This points to a bad image tag, a private registry with missing credentials, or a network issue reaching the registry.
A status of Pending with no transition to Running usually means the pod cannot be scheduled — most often because no node has enough available CPU or memory to accommodate it, or because a node selector or affinity rule is too restrictive.
A status of Error is a catch-all that typically means the container ran and exited with a non-zero exit code, which signals an application-level failure.
The RESTARTS column in kubectl get pods output is equally informative. A high restart count paired with CrashLoopBackOff is a strong signal that the application is consistently failing at startup rather than occasionally crashing under load.
Describing the Pod: Your First Deep Dive
Once you have identified a problem pod, the next step is running kubectl describe pod followed by the pod name and namespace. This command gives you the most comprehensive single view of a pod's state and history available from the control plane.
The Events section at the bottom of the describe output is where most of the diagnostic gold is. It shows you exactly what Kubernetes did and when — image pulls, scheduling decisions, container starts, liveness probe failures, OOMKill events, and more. Always read the events section first before diving into logs.
The Conditions section tells you whether the pod has been scheduled, whether its init containers completed successfully, and whether it is considered ready to serve traffic. A pod that is Running but not Ready usually means a readiness probe is failing, which means traffic is being withheld even though the container is technically up.
The Containers section shows you the image being used, the resource requests and limits configured, the environment variables injected, and the last termination state of the container — including the exit code and the reason for termination. That last termination state is particularly valuable for diagnosing what happened in the previous crash before logs were lost.
Reading Pod Logs: What to Look For
Logs are usually where you find the actual error message that caused the crash. Running kubectl logs followed by the pod name gives you the current container's stdout and stderr output.
The most important flag to know is --previous, which retrieves the logs from the last terminated container instance rather than the current one. In a CrashLoopBackOff situation, the current container may have barely started before crashing, giving you almost no log output. The --previous flag lets you see what the container printed before it died, which is often where the real error message lives.
For multi-container pods, use the -c flag to specify which container's logs you want to read. Init containers are a common source of silent failures — the main container never starts because an init container failed, but if you only check the main container's logs you will see nothing useful.
If the application crashes quickly and you need to capture logs in real time, the -f flag streams logs live. Combine this with a manual restart or a fresh deployment to catch the startup sequence as it happens.
What you are looking for in logs varies by application, but the most common signals are missing environment variables referenced by name, failed database or service connections, permission errors reading mounted files or secrets, and panic stack traces from application runtimes like Go, Java, or Node.js.
Decoding Exit Codes
The exit code of a crashed container is one of the most direct signals available to you. Kubernetes surfaces the last exit code in the kubectl describe pod output under the Last State section.
Exit code 0 means the process exited cleanly — which in the context of a crashing pod usually means the application intentionally shut itself down, possibly due to a failed health check or a graceful shutdown triggered by a signal.
Exit code 1 is a generic application error. The application ran but encountered a fatal condition and exited. The logs will have the details.
Exit code 137 means the process was killed by signal 9, which is SIGKILL. In Kubernetes, this almost always means the container was OOMKilled — the kernel terminated it because it exceeded its memory limit.
Exit code 139 means a segmentation fault — the process tried to access memory it was not allowed to access. This is typically a bug in native code or a corrupted binary.
Exit code 143 means the process received SIGTERM and exited gracefully. This is normal during rolling updates or pod evictions, but if you are seeing it repeatedly it may indicate a liveness probe is timing out and Kubernetes is terminating the pod.
Exit codes in the 128+ range generally represent fatal signal-based terminations. Exit code 128 plus the signal number gives you the signal that caused the termination.
Diagnosing CrashLoopBackOff in Detail
CrashLoopBackOff deserves its own section because it is the most common and most frustrating pod status engineers encounter. The pattern is always the same: the container starts, something goes wrong almost immediately, it exits, Kubernetes restarts it, and the cycle repeats with exponentially increasing wait times between restarts.
The most productive debugging sequence for CrashLoopBackOff is to first check kubectl describe pod for the last termination exit code and reason, then check kubectl logs with the --previous flag to read the final output before the last crash, and then look carefully at the environment variables, volume mounts, and image tag shown in the describe output.
The most common causes of CrashLoopBackOff in order of frequency are: a missing or incorrectly named environment variable the application requires to start, a secret or configmap that was referenced in the pod spec but does not exist in the namespace, an application that cannot connect to a dependency like a database on startup and fails hard instead of retrying, an incorrectly set command or entrypoint in the container spec that causes the process to exit immediately, and an application that panics on startup due to a code bug introduced in the latest deployment.
Debugging with an Ephemeral Debug Container
Sometimes the container image your application runs in is so minimal — a distroless or scratch image — that it has no shell, no curl, no ping, and no diagnostic tools whatsoever. In these situations you cannot exec into the container to investigate from the inside.
Kubernetes 1.23 introduced ephemeral containers as a stable feature specifically for this scenario. An ephemeral container is a temporary container you inject into a running pod for debugging purposes. It shares the pod's network namespace and can optionally share the process namespace, letting you run diagnostic tools against the running application from a rich debug image without modifying the original pod spec.
This is particularly powerful for diagnosing networking issues inside a pod — you can run DNS lookups, test TCP connections, check which ports are actually open, and verify that service endpoints are reachable, all from inside the pod's network context.
Checking Resource Limits and OOMKill
If your pod is being OOMKilled, the debugging path is straightforward but the fix requires some analysis. Start by checking the memory limit configured in the pod spec via kubectl describe pod. Then look at actual memory consumption using kubectl top pod, which requires the metrics server to be installed in your cluster.
If the pod's memory usage was approaching or exceeding the limit before the kill, you have a few options: increase the memory limit if the application genuinely needs more, investigate the application for memory leaks if consumption grows unboundedly over time, or tune application-level memory settings like JVM heap size for Java applications.
If OOMKill is a recurring issue rather than a one-off event, it may be a signal that your workload needs dynamic resource management. Understanding Kubernetes autoscaling with HPA and VPA can help you right-size pods automatically based on actual consumption rather than static guesses, which is a far more sustainable solution than manually bumping memory limits every few weeks.
OOMKill situations are also worth checking at the node level. If multiple pods on the same node are being killed, the node itself may be under memory pressure and the kubelet may be evicting pods to protect system stability, even if individual pods are within their limits.
Investigating Liveness and Readiness Probe Failures
Liveness and readiness probes are the mechanisms Kubernetes uses to know whether your application is healthy. A liveness probe failure causes Kubernetes to kill and restart the container. A readiness probe failure removes the pod from service endpoints so it stops receiving traffic, but does not restart it.
Both types of probe failures show up clearly in the Events section of kubectl describe pod. If you see repeated "Liveness probe failed" events, the question to answer is whether the probe is correctly configured or whether the application is genuinely unhealthy.
Common probe misconfiguration issues include an initialDelaySeconds that is too short, causing Kubernetes to start probing before the application has finished its startup sequence. Another common issue is a probe that checks an endpoint which requires the application to be fully initialized, when in reality the application takes 30 to 60 seconds to warm up. Increasing the initialDelaySeconds or using a separate startupProbe to handle slow-starting applications resolves these cases.
If the probe configuration looks correct and the application is genuinely failing the health check, the probe failure is a symptom rather than the root cause. Dig into application logs to understand why the health endpoint is returning failures.
Debugging Image and Registry Issues
ImagePullBackOff is one of the simpler failure modes to diagnose but still trips up teams regularly. The describe pod events will tell you exactly which image tag it tried to pull and what error it received.
The most common causes are a typo in the image tag, pushing an image to a registry under a different tag than what the pod spec references, using the latest tag in a production environment where the registry was updated with a broken image, and missing imagePullSecrets for private registries. Teams using Helm charts for Kubernetes deployments sometimes misconfigure image tags inside their values files, causing ImagePullBackOff errors that are easy to overlook during a rushed release.
For private registries, Kubernetes needs a Secret of type docker-registry in the same namespace as the pod, referenced in the pod spec via imagePullSecrets. A missing or expired registry credential is a frequent culprit for ImagePullBackOff in production clusters.
Using kubectl exec for Live Investigation
When a pod is running but misbehaving — perhaps it started successfully but is producing errors or unexpected behavior — kubectl exec lets you open an interactive shell inside the running container. From there you can inspect the filesystem, check environment variables, test network connectivity to other services, and run application-specific diagnostic commands.
This is particularly useful for verifying that secrets and configmaps are being mounted correctly, that file permissions on mounted volumes are what the application expects, and that the application can actually reach its dependencies from within the pod's network namespace. If you are new to kubectl or want to make sure you are using it to its full potential, going through a thorough kubectl installation and configuration guide will save you a significant amount of time when you are in the middle of a live incident.
Debugging Init Container Failures
Init containers run to completion before the main application container starts. If an init container fails, the main container never runs and the pod stays in an Init state. This is easy to miss if you only check the main container's logs.
To debug an init container failure, use kubectl describe pod to see the state of each init container listed under the Init Containers section. Then use kubectl logs with the -c flag and the init container's name to read its output. Common init container failures include waiting for a dependency that never becomes available, a permission error when setting up a volume, and a misconfigured script that exits with a non-zero code.
Pod-Level Debugging Is Only Part of the Story
Pod-level debugging tells you what went wrong inside a container, but it does not always tell you why it went wrong at the infrastructure level. If a pod is being denied access to a secret or configmap and crashing as a result, the root cause may actually be a permissions issue rather than an application bug. Brushing up on Kubernetes RBAC and Role-Based Access Control will help you rule out authorization failures as a root cause before you spend an hour staring at application logs that point nowhere useful.
Building a Systematic Debugging Habit
The biggest mistake engineers make when debugging crashing pods is jumping straight to assumptions — "it must be the database connection" or "we probably pushed a bad image" — without first gathering evidence. The commands covered in this guide take less than two minutes to run through systematically, and they almost always point you directly at the root cause.
A reliable sequence to follow every time is: check pod status and restart count with kubectl get pods, read the events and last termination state with kubectl describe pod, check the previous container logs with kubectl logs --previous, verify resource consumption with kubectl top pod, and only then form a hypothesis about the root cause and begin making changes.
Kubernetes gives you more observability into application failures than most platforms ever have. The engineers who resolve incidents fastest are not the ones who know the most obscure kubectl flags — they are the ones who have internalized a calm, systematic approach and follow it every time regardless of the pressure they are under.
If you want to truly master Kubernetes debugging, incident response, and production cluster operations, a structured DevOps Online Course at Paperlive gives you the hands-on experience to build this muscle memory in a safe environment before you are under real production pressure. The curriculum is built around real cluster scenarios, not toy examples.
Beyond kubectl: When You Need More
For persistent or hard-to-reproduce issues, basic kubectl commands may not be enough. Integrating proper observability tooling is non-negotiable for production clusters. A tool like Grafana paired with Prometheus gives you historical visibility into resource usage, crash patterns, and alert thresholds that kubectl alone can never provide. If you are not already using it, the complete guide to Grafana dashboards and observability is a natural next step after you have your debugging workflow locked in.
Cluster-level audit logs can also reveal whether a pod was deleted externally, whether a secret it depended on was modified, or whether a resource quota was hit — none of which are visible in pod-level logs alone.
The best time to set up observability is before you have a production incident. The second best time is immediately after your first one.
Debugging crashing pods is a skill that compounds over time. Every incident you diagnose carefully teaches you something about how Kubernetes, your applications, and your infrastructure interact. If you want to accelerate that learning with structured hands-on labs, real cluster scenarios, and expert guidance, DevOps Training Online at Paperlive covers the full spectrum — from kubectl fundamentals to production incident response — so you build genuine expertise rather than just familiarity with commands.
FAQs: Debugging Crashing Pods in Kubernetes
What does CrashLoopBackOff mean in Kubernetes?
CrashLoopBackOff means a container started, crashed, and Kubernetes is repeatedly trying to restart it with progressively longer wait times between attempts. It almost always indicates the application is failing on startup. Check the previous container logs using kubectl logs --previous and the last termination exit code in kubectl describe pod to find the root cause.
How do I see why a Kubernetes pod crashed?
Start with kubectl describe pod to read the Events section and last termination state, then use kubectl logs --previous to retrieve logs from the container before it crashed. The exit code surfaced in the describe output is also a key signal — exit code 137 means OOMKill, exit code 1 means a generic application error, and exit code 143 means the process received SIGTERM.
What is OOMKilled in Kubernetes?
OOMKilled means the container exceeded its configured memory limit and was forcibly terminated by the Linux kernel. The fix is either to increase the memory limit in the pod spec, investigate the application for memory leaks, or tune runtime memory settings for applications like the JVM. You can verify memory consumption using kubectl top pod.
Why is my Kubernetes pod stuck in Pending?
A pod stuck in Pending usually means it cannot be scheduled onto any node. The most common reasons are insufficient CPU or memory available across the cluster's nodes, a node selector or affinity rule that no current node satisfies, or a taint on all nodes that the pod does not tolerate. Check the Events section in kubectl describe pod for the specific scheduling failure reason.
How do I debug a pod with no shell in its container image?
If your container image is minimal and has no shell or diagnostic tools, use Kubernetes ephemeral containers — available as a stable feature since Kubernetes 1.23 — to inject a temporary debug container into the running pod. The ephemeral container shares the pod's network namespace, letting you run diagnostic tools against the application from a full-featured debug image without modifying the original pod spec.
