Kubernetes events are not primarily designed for incident response. They are designed for operator feedback: things like Pulled image, Created container, Started. But mixed into that stream of routine status events are signals that, taken together, describe exactly what happened during a failure. The problem is that nobody is reading them that way.
Most monitoring setups treat Kubernetes events as low-priority auxiliary data. Metrics from kube-state-metrics and node exporters drive the alert rules. Logs drive the investigation. Events get queried after the fact if the other sources do not explain the incident. This is backwards. Kubernetes events are often the earliest and most precise signal of a developing failure, and they contain correlation information that metrics and logs require significant engineering to reconstruct.
This article is about which event patterns actually matter and how they correlate across failure modes.
The Kubernetes Event Data Model
Events in Kubernetes are first-class API objects, queryable via kubectl get events or the API at /api/v1/events. Each event has a reason field (a short machine-readable string like OOMKilling or FailedScheduling), a message field (human-readable context), an involvedObject that identifies the resource the event concerns, and a firstTime plus lastTime plus count structure that handles repeated events.
The count field is important and frequently misread. A single Event object with count: 47 does not mean 47 separate events. It means the same event with the same message and reason was deduplicated 47 times. This matters for correlation: you want to know the frequency of a repeated event, not just whether it occurred. The kubelet deduplicates within a 10-minute window, so a count of 47 over two hours suggests the condition is continuous, not just a transient spike.
CrashLoopBackOff: The Event Chain Before the Alert Fires
By the time an alert fires on kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}, a predictable chain of events has already happened. Understanding this chain tells you whether you are looking at a code crash, an OOM kill, or a misconfigured readiness probe.
The typical event sequence for a container that crashes on startup:
reason: BackOff
message: Back-off restarting failed container
count: increasing
involvedObject: Pod/api-worker-5d9f6-abc12
reason: Failed
message: Error: failed to create containerd task:
failed to create shim: OCI runtime create failed
involvedObject: Pod/api-worker-5d9f6-abc12
This sequence, Failed followed by growing BackOff count, is a container that cannot start at all. The fix direction is almost always the image or the entrypoint.
A different sequence points to a different root cause:
reason: Started
message: Started container api-worker
involvedObject: Pod/api-worker-5d9f6-abc12
reason: OOMKilling
message: Memory limit reached, killed process
involvedObject: Node/prod-node-3
reason: BackOff
message: Back-off restarting failed container
involvedObject: Pod/api-worker-5d9f6-abc12
Here the container starts successfully but the kubelet on the node emits an OOMKilling event against the node object before the pod enters BackOff. The fix direction is resource limits. The key insight: the OOMKilling event is on the Node object, not the Pod object. If you are only watching pod events, you will see BackOff with no visible cause. You need to correlate across the involvedObject axis to find the OOM signal.
FailedScheduling and Node Pressure: The Resource Starvation Chain
When pods cannot schedule, the Kubernetes scheduler emits FailedScheduling events with messages that contain the exact reason. These are among the most diagnostic events in the system because the scheduler's message is precise about which constraint failed.
reason: FailedScheduling
message: 0/5 nodes are available:
2 node(s) had taint {node.kubernetes.io/
memory-pressure: }, that the pod didn't
tolerate. 3 node(s) were unschedulable.
involvedObject: Pod/checkout-7b8c9-xyz99This message tells you that two nodes are under memory pressure and three are cordoned or unschedulable. Neither of those conditions requires you to look at metrics first. The event tells you both the failure mode and the scope.
The correlation point: when you see FailedScheduling for multiple pods in quick succession across different namespaces, the root cause is almost always at the node or cluster level, not at the pod level. A single node under memory pressure generates MemoryPressure node condition events, Evicted pod events for the pods it evicts to free memory, and FailedScheduling events for new pods that try to land on that node. These three event types across three different object kinds all trace back to the same node.
A cluster where one node enters memory pressure generates event signatures across pods, nodes, and the scheduler simultaneously. Reading any one of those in isolation misses the shared root cause.
Readiness Probe Failures and Traffic Shedding
Readiness probe failures are often underweighted in alert triage because they do not immediately cause container restarts. A pod in a Running state with failing readiness probes is still running. It is just removed from the Service endpoint set, which means it is shedding traffic without appearing in a CrashLoopBackOff alert.
reason: Unhealthy
message: Readiness probe failed: Get
http://10.0.4.22:8080/healthz: context
deadline exceeded (Client.Timeout exceeded)
involvedObject: Pod/order-svc-6f7g8-mno45By itself this is ambiguous. The probe failure could be because the container is genuinely unhealthy, or because it is overloaded and too slow to respond to the probe, or because the probe endpoint path is wrong.
The correlating event is on the Endpoints object:
reason: UpdatedLoadBalancer
message: Updated load balancer with new hosts
involvedObject: Service/order-svc
When you see Unhealthy on pods followed by UpdatedLoadBalancer (or its equivalent in ingress-controller events) on the Service, you know traffic routing has changed as a result of the probe failures. This is a user-facing impact, not just an operational condition. The correlation between the pod-level probe event and the service-level routing change is the signal that this is worth escalating from background noise to active investigation.
HPA Thrashing and Metric Lag
HPA events are another underused source. When the HPA is working correctly, you see occasional SuccessfulRescale events. When it is not, the event stream is diagnostic:
reason: FailedGetResourceMetric
message: unable to get metrics for resource
cpu: unable to fetch metrics from
resource metrics API: the server is
currently unable to handle the request
involvedObject: HorizontalPodAutoscaler/
worker-hpa
This event tells you that the HPA cannot get metrics from the metrics server. It will not scale. Any pods that are overloaded will stay overloaded until the metrics server recovers. The correlation: if you are seeing latency alerts on the pods governed by this HPA at the same time as FailedGetResourceMetric HPA events, the latency is at least partly because the HPA cannot scale to absorb load.
HPA thrashing, where the HPA scales up and down rapidly, appears as a sequence of SuccessfulRescale events with alternating reason messages. This is usually a metric lag problem: the HPA acts on metrics that lag actual traffic by enough that by the time the scale-up pods are running, the traffic spike has passed and the metric has dropped, triggering a scale-down. Correlating the HPA event timestamps with the metric query window in the HPA spec reveals whether the lag is within or outside the configured stabilization window.
Event Cardinality and Retention: The Practical Problem
Kubernetes events have a default retention of one hour in the API server, and most clusters have a limit on the total number of event objects (configurable but typically around 4000). In a cluster under load, event objects get evicted before they can be read, especially for the high-frequency events like BackOff that repeat continuously during a CrashLoop.
This means that for event-based correlation to work reliably, you need to ship events out of the API server to an external store before they expire. Kubernetes Event Exporter or a simple controller that watches the events API and writes to your log aggregation pipeline are the common approaches. The events themselves are small objects. Shipping them all is not a storage problem, it is an operational setup problem that most teams skip.
The result of skipping this setup: events are useful for immediate incident response, but useless for post-incident analysis of anything that happened more than an hour ago. Most post-incident reviews happen more than an hour after the incident. This is a solvable problem, but it requires investment in the observability pipeline before the incident, not during it.
What Good Event Correlation Looks Like in Practice
The failure modes described above are not exotic. CrashLoopBackOff from OOM, node pressure cascading to scheduling failures, readiness probe failures causing traffic shedding, HPA metric failures during load spikes: these are the incidents that happen in production clusters running real workloads. The events that describe them exist in the API server today.
The gap is not in the data. It is in tooling that reads events across the involvedObject dimension, correlates by time window, and groups by shared root cause. An alert that says "pod order-svc-6f7g8-mno45 is in BackOff" is less useful than "pod order-svc-6f7g8-mno45 entered BackOff after an OOMKilling event on prod-node-3, which also generated FailedScheduling for 4 other pods in the last 15 minutes." The second form does not require an investigation. It hands you the conclusion.
We are not claiming that this is always simple to build. Cross-kind correlation with timestamp windowing is real engineering work. But the input data is available in a standard Kubernetes cluster without any additional instrumentation. The events are already there. The question is whether your tooling is reading them.