From Alert to Fix: A Walk Through an SRE Investigation at 3am

This is a walkthrough of a composite incident drawn from patterns we have seen repeatedly when talking with SRE teams. The details are illustrative rather than a specific customer case, but the steps, the dead ends, and the timing are representative of real investigations. The goal is to show concretely what the investigation process looks like minute by minute, and to identify which steps add value versus which steps consume time without advancing the diagnosis.

3:07am: The page fires

The PagerDuty notification says: HighErrorRate: order-service error rate above 5% for 5m. The on-call engineer, call them the engineer, wakes up, opens the laptop. The first question is always: is this a real user-facing problem or a metric blip? Error rate at 5% for 5 minutes on order-service is real enough to investigate.

The engineer opens the alerting dashboard. There is one active alert: the order-service error rate. A quick check of the service status page confirms that order processing is degraded. Orders are failing for a portion of requests. Not all requests, not zero requests: roughly 1 in 20 is failing.

This partial failure pattern is more ambiguous than a total outage. A total outage often has an obvious cause: the pod is down, the node is gone, the database is unreachable. Partial failures require more investigation because they can come from many places: a pod with a memory pressure issue that is processing some requests normally and failing others when it hits memory ceiling, a downstream dependency that is timing out intermittently, a bad deployment that introduced a code path that fails for specific request shapes, or load-balancing routing a portion of traffic to a pod in a bad state.

3:11am: Starting the kubectl investigation

The engineer runs the first set of commands. Check pod status in the order-service namespace:

kubectl get pods -n order-service \
  -o wide
NAME                           READY   STATUS
order-svc-7f9b4c2d8-k4xmp      1/1     Running
order-svc-7f9b4c2d8-t8wlq      1/1     Running
order-svc-7f9b4c2d8-rzn6p      1/1     Running

All three pods are Running and Ready. No CrashLoopBackOff. No pending pods. No obvious signal from pod status. This is the first dead end: the pods look healthy but the service is not performing correctly. The engineer now has to decide where to look next.

The two most common next steps are: check pod logs for recent errors, or check recent events for the namespace. Both are reasonable, and experienced engineers have intuitions about which yields signal faster for different failure modes. At 3am, with cognitive function not at its peak, the decision is often made somewhat arbitrarily.

3:14am: Pod logs, the first dead end

The engineer checks logs for one pod:

kubectl logs \
  order-svc-7f9b4c2d8-k4xmp \
  --tail=100 \
  -n order-service

The logs show normal request processing with occasional error lines. The errors all have a consistent pattern: timeout errors calling an upstream service. The upstream service name in the error is payment-processor. So the errors in order-service are caused by timeouts calling payment-processor. The engineer checks the other two pods and sees the same pattern.

This looks like the root cause: payment-processor is slow or failing, causing order-service to time out. But before changing anything, the engineer needs to understand why payment-processor is having problems. Is this a payment-processor issue, or is payment-processor having problems because something further upstream is causing it to fail?

3:21am: Chasing the dependency chain

The engineer checks payment-processor:

kubectl get pods -n payment-service \
  -o wide
NAME                              READY   STATUS
payment-proc-5c8d3f6b9-w2n4k      0/1     CrashLoopBackOff
payment-proc-5c8d3f6b9-m9xqr      1/1     Running
payment-proc-5c8d3f6b9-v7p5s      1/1     Running

One of three payment-processor pods is in CrashLoopBackOff. The other two are Running. The HPA is attempting to maintain three ready replicas, but with one in CrashLoopBackOff, the load-balancing is spreading traffic across only two healthy pods. At high enough request volume, two pods cannot absorb the load, and their own latency increases. Upstream callers like order-service start seeing elevated latency and eventually timeouts.

This explains the partial failure pattern. If the traffic were evenly distributed across three pods and one pod is in CrashLoopBackOff, roughly one-third of requests would be failing. The actual error rate is around 5%, not 33%, which suggests the Kubernetes service load balancing and connection pooling are absorbing some of the failure by routing away from the unhealthy pod, but not all of it.

Now the engineer knows: the origin is in payment-processor. The CrashLoopBackOff on one pod is causing elevated latency on the healthy pods, which is causing order-service timeouts, which is generating the high error rate alert. The investigation has found the source. The next step is understanding why the payment-processor pod is crashing.

3:27am: Finding the specific failure

The engineer checks the describe output for the crashing pod:

kubectl describe pod \
  payment-proc-5c8d3f6b9-w2n4k \
  -n payment-service

The Events section of the describe output shows:

Events:
  Warning  BackOff  3m  kubelet
    Back-off restarting failed container
  Warning  OOMKilled  8m  kubelet
    Container killed due to OOMKilled
  Warning  OOMKilled  14m  kubelet
    Container killed due to OOMKilled

OOMKilled twice in the last 14 minutes. The container is running out of memory and being killed by the kubelet. The engineer checks the pod's resource limits:

kubectl get pod payment-proc-5c8d3f6b9-w2n4k \
  -n payment-service \
  -o jsonpath='{.spec.containers[0].resources}'

# limits:   cpu=500m  memory=256Mi
# requests: cpu=100m  memory=128Mi

Memory limit is 256Mi. The engineer checks current memory usage across the healthy pods to understand whether 256Mi is the right limit for current traffic:

kubectl top pods -n payment-service
NAME                              CPU     MEMORY
payment-proc-5c8d3f6b9-m9xqr      340m    218Mi
payment-proc-5c8d3f6b9-v7p5s      320m    224Mi

The two healthy pods are using 218Mi and 224Mi of memory respectively, both very close to the 256Mi limit. This confirms the diagnosis: the current traffic load is pushing memory usage to near the limit on all pods. The one pod that OOMKilled was likely handling a memory-intensive request batch when it crossed the 256Mi ceiling. The other two are also close to their limit; at the current request volume, more OOMKills are likely in the near term.

The investigation took 20 minutes to trace from an order-service error rate alert back to a memory limit set on a payment-processor pod. Each hop added delay without adding complexity: the chain was linear once you found the first link.

3:31am: The fix decision

The diagnosis is clear. The proposed fix is raising the memory limit for the payment-processor Deployment from 256Mi to at least 512Mi, and also increasing the memory request from 128Mi to something that better reflects actual usage (at least 200Mi) to ensure the scheduler places the pod on a node with sufficient memory headroom.

Before applying the fix, the engineer spends a few minutes considering whether there are any reasons not to do this right now. Has there been a recent deployment to payment-processor? No, the last deployment was four days ago and has been running fine. Is there a known memory leak? Not according to the runbook. Is the current memory usage genuinely higher than historical baseline, suggesting a traffic spike rather than a misconfigured limit? The engineer pulls up the memory metric for payment-processor over the last 24 hours and sees a gradual increase over the past 8 hours correlated with a business event. The traffic is genuinely higher and the limit is genuinely too low. The fix is appropriate.

The engineer edits the Deployment:

kubectl edit deployment \
  payment-proc \
  -n payment-service

Changes the memory limit from 256Mi to 512Mi and the memory request from 128Mi to 256Mi. Saves the edit. The Deployment controller triggers a rolling update. New pods come up with the higher limit. The CrashLoopBackOff pod terminates and is replaced by a healthy pod with the new limits.

3:38am: Verification

The engineer waits for the rolling update to complete and then verifies:

kubectl get pods -n payment-service \
  -o wide
NAME                              READY   STATUS
payment-proc-6a4b2e7c1-n8kxw      1/1     Running
payment-proc-6a4b2e7c1-p2mrz      1/1     Running
payment-proc-6a4b2e7c1-q5vt8      1/1     Running

All three pods running. The engineer checks the alerting dashboard: the order-service error rate alert is no longer firing. The error rate has dropped back to near-zero. Order processing is normal.

The incident is resolved. The engineer writes up a brief incident note and goes back to sleep at 3:51am. Elapsed time from first page: 44 minutes.

Where better tooling would have mattered

The investigation had three distinct phases: finding the alerting service (order-service), tracing back to the origin service (payment-processor), and confirming the specific failure and its fix (OOMKilled pod with a low memory limit). Each phase took between 6 and 10 minutes. The total elapsed time was around 44 minutes, which is not catastrophic but is a substantial amount of cognitive effort at 3am.

The first phase consumed 14 minutes and involved a dead end: checking order-service pod status and logs before realizing that order-service was the victim, not the cause. That dead end was predictable: the high error rate alert fired on order-service because order-service is where the end-user-visible failures occurred, not because order-service itself was the root cause. An event-correlation system that reads the timeline of Warning events across namespaces and surfaces the earliest Warning event would have pointed at payment-processor immediately, skipping the 14-minute order-service investigation detour.

The second phase (tracing payment-processor OOMKills) would have been fast regardless: once you know to look at payment-processor, the kubectl describe output is unambiguous. This is the part where tooling adds the most value with the least risk: the event data needed for diagnosis is right there in the cluster, and presenting it in structured form is purely informational, not action-taking.

The third phase, the fix decision, required the engineer's contextual knowledge. No tooling can know whether the current memory usage increase is a traffic spike, a leak, or a permanent capacity expansion. The engineer confirmed it was a traffic spike by pulling historical metrics. A diagnostic tool that proposed the memory limit increase as the fix would be correct in this case. But the engineer's decision to apply it, informed by the historical metrics check, is the moment where human judgment was genuinely necessary and could not have been safely replaced.

The approval gate in that scenario is the right model: show the engineer the diagnosis (OOMKilled, memory at 218-224Mi against a 256Mi limit), show the proposed fix (raise limit to 512Mi, raise request to 256Mi), and wait for the engineer to approve. The engineer's contribution in the approval step is real and takes less than two minutes when the diagnosis is clear. The value is not in removing that two minutes. It is in replacing the 20 minutes of investigation that preceded it.