Introduction

When a Kubernetes pod fails, it is usually quite easy to figure out why it died. You can just check the logs for a pod crash-looping, or verify the exit status and reason. But this is not always true.

I ran into a good example of this with K8ssandra. In a K8ssandra pod the Cassandra management API is PID 1, and it starts Cassandra as a child process, so the process the kubelet is watching is the management API rather than the database. When Cassandra dies, what you see in the pod status is whatever the wrapper decided to report on its behalf, which is often not the same thing as what actually happened.

That is not really a Cassandra problem, nor is it a dig at K8ssandra. It is a property of how containers work, and once you recognise the pattern, you start finding it all over the place.

What PID 1 means to Kubernetes

A container is a cgroup and a set of namespaces with one process at the root, and everything the kubelet reports about that container is derived from that single process: the exit code you see is the exit status of PID 1. The restart happens because PID 1 exited, and Reason: OOMKilled appears when the container cgroup reports an out-of-memory event and PID 1 dies as a result of it. The termination message is read from /dev/termination-log, so it says whatever the container chose to write there, and nothing at all if it wrote nothing. On shutdown, SIGTERM is delivered to PID 1 and nowhere else.

If your workload is PID 1, all of that works nicely and the pod status means what it appears to mean. If it is not, you can't know for sure what wrote to the log last, or the real cause of the exit.

When PID 1 is not your workload, four things go wrong

The child dies and the parent carries on

The worst case is the one where nothing happens at all. The pod stays Running, no restart is triggered, no event is recorded, and there is nothing in kubectl describe to look at. The only thing that will tell you the workload has been dead for hours is the liveness probe, and that only helps if the probe asks the workload a question, rather than checking that the supervisor is still answering its own port. An endpoint that falsely returns 200 because the wrapper is fine while the application behind it is gone is worse than having no probe at all, because it hides a problem that could otherwise have been caught easily by your monitoring.

The exit code you get belongs to the wrapper

Slightly better, though still lossy, is the case where the supervisor notices the child has gone, tidies up after it, and then exits on its own terms. In an issue I dealt with recently, Cassandra was killed with exit code 137, the wrapper exited with code 1, and Kubernetes recorded an Error with exit code 1. As a result, the 137 was lost, and the OOM attribution was lost with it. I wasted time diagnosing the wrong problem with a customer instead of the real one, the memory.

SIGTERM never reaches the workload

Shutdown has the same problem in reverse. When the kubelet evicts a pod or drains a node, it sends SIGTERM to PID 1 and starts the grace-period clock. A wrapper invoked through sh -c will not forward that signal to anything, so the workload carries on unaware until SIGKILL arrives at the end of the grace period. On a stateless web service you lose a few connections in flight and nobody notices. On anything that holds state you get an unclean shutdown with no flush and no drain, followed by a restart with recovery work to do: fine once, considerably less fine when you are rolling a StatefulSet one pod at a time.

Zombies

The last one is quieter and takes longer to surface. PID 1 in a namespace inherits orphaned children and is expected to reap them, which application runtimes generally do not implement, because nobody wrote them expecting to be PID 1. If anything in the container forks (a backup job, a nodetool invocation, a health script), defunct entries accumulate in the process table until something eventually breaks in a way that looks nothing like the actual cause.

Where else this shows up

Running multiple applications within a pod is not recommended. As you can see, this brings its own set of problems, but sometimes it is unavoidable. I bring up the K8ssandra example because it was the latest to bite me, but you'll find this elsewhere. It is worth knowing what PID 1 actually is in the containers you run. The usual candidates:

• An operator that ships a management or agent process inside the workload container rather than beside it.
• supervisord, s6 or runit in an image, usually because two processes needed to share a filesystem or a localhost port.
• An entrypoint script ending in myapp instead of exec myapp, which leaves the shell as PID 1 for the life of the pod.
• An entrypoint that backgrounds the real process and then calls wait.
• A runtime that forks workers, such as a gunicorn master, a php-fpm pool, or Node in cluster mode.
• A shim that fetches secrets or waits for a dependency before handing over to the real thing.

Finding out takes a few seconds:

# run inside the pod's container
kubectl exec POD -c CONTAINER -- ps -eo pid,ppid,comm

If PID 1 is not the thing whose name is on the deployment, the rest of this applies to you.

Fixing it properly means giving each process its own PID 1

The real fix is not to put two processes in one container in the first place. Native sidecar containers, init containers carrying restartPolicy: Always, went stable in Kubernetes 1.29 and cover most of what people reached for supervisord to do. Because each process then gets its own PID 1, it also gets its own exit code, its own OOM accounting, and its own restart behaviour. Everything below this paragraph is mitigation for when you cannot restructure the pod, which with somebody else's operator is most of the time.

Where you do own the image, the cheapest win by a very wide margin is putting exec in front of the final command in the entrypoint script. This is a simple fix that works well for many apps: exec replaces the running process and acquires PID 1 in its place.

Where the container genuinely needs an init process, use something written to be one: tini (my preferred option) or dumb-init as PID 1 costs you nothing and gives you signal forwarding and zombie reaping. Docker's --init flag and the Kubernetes shareProcessNamespace option are variations on the same idea.

If you are stuck with a supervisor, at least make it an honest one. A wrapper that owns a workload should forward SIGTERM and SIGINT to the child, propagate the child's exit code verbatim rather than substituting one of its own, and write a line to /dev/termination-log on the way out. Exit-code fidelity is the part that preserves the 137, and it is usually a handful of lines in whatever language the wrapper happens to be written in. While you're in there, point the liveness probe at something only a working workload can answer: a probe against the supervisor's own port is precisely what turns a crash into a silent outage.

Making the failure visible when you cannot fix it

Setting terminationMessagePolicy: FallbackToLogsOnError on the container is one line of YAML and does more than almost anything else on this list. With it set, the kubelet copies the tail of the container log into the termination message whenever a container exits non-zero without writing to /dev/termination-log itself, putting the last words of the process directly into kubectl describe, where the engineer is already looking. Very few people set it.

Beyond the pod, alert on kube_pod_container_status_last_terminated_reason from kube-state-metrics rather than relying on events, since the metric carries the reason as a label and survives long after the event has aged out of the API server. A restart with a reason of Error on a workload that should never crash is worth an alert on its own, quite separately from whatever the application dashboards are saying.

When the kubelet view is ambiguous, dmesg is not. Running node-problem-detector on your nodes reads the kernel ring buffer and surfaces OOM kills as node conditions and Kubernetes events, often the only place you'll find out which process the kernel actually picked, which is the question the pod status declined to answer in the first place.

Cgroups and the OOM killer decide which failure mode you get

This is worth knowing because it decides which of the failure modes above you get. Under cgroup v1 the kernel OOM killer picks a single victim, and that victim can be your workload while PID 1 carries on running: the silent case, delivered by the kernel rather than by a supervisor. Cgroup v2 added memory.oom.group, which kills every process in the cgroup as a unit, and container runtimes on cgroup v2 set it for containers, so an out-of-memory event takes the whole container down and you get a clean OOMKilled with exit 137, no matter which process crossed the line. Kubernetes has since added a singleProcessOOMKill kubelet option to bring the old behaviour back for workloads that relied on it, so it is worth checking what your nodes actually do rather than assuming:

# cgroup2fs on v2, tmpfs on v1
stat -fc %T /sys/fs/cgroup/

Back to Cassandra

All of the above comes before you reach for JVM flags, but the Cassandra-specific hardening is worth having once the structural work is done. Adding -XX:+ExitOnOutOfMemoryError to jvm-options on the CassandraDatacenter makes the JVM die immediately instead of limping along in a state where it still answers health checks and serves nothing useful. Pairing it with -XX:+HeapDumpOnOutOfMemoryError and a heap dump path on persistent storage means the evidence outlives the pod rather than disappearing with it:

# jvm-options on the CassandraDatacenter
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/path/on/persistent/storage

Sizing matters as much as the flags. Keep -Xmx well below the container limit, because Cassandra's off-heap use covers memtables, bloom filters, compression metadata and direct buffers, and the management API wants its own footprint on top of all that. Two thirds to three quarters of the limit is a sensible place to start before you tune from what you observe. Setting requests.memory equal to limits.memory puts the pod into Guaranteed QoS, which gives it a better oom_score_adj and keeps it out of node-level eviction races that have nothing to do with Cassandra at all.

Where to start

If I had to reduce all of this to a few lines:

• Find out which process is PID 1 in every container you run in production.
• Put exec in the entrypoints you own.
• Prefer a native sidecar to a supervisor whenever the pod is yours to restructure.
• Set terminationMessagePolicy: FallbackToLogsOnError across the board.
• Get node-problem-detector onto the nodes, so the kernel's version of events is written down somewhere you can reach at three in the morning.

None of it is difficult, and most of it is decided long before the container ever starts, which is exactly why it is worth getting right early. A process that dies quietly is not a smaller problem than one that dies loudly. It is just one you find out about later, usually from a customer rather than from a pager.

I, for one, welcome our new robot overlords.