When we started building NudgeBee, the first design question was the hardest one: should the system act on its diagnoses autonomously, or should it wait for a human to approve each proposed change? The efficiency argument for autonomous action is compelling on paper. If the system can correctly identify that payments-api needs its memory limit raised, and it can generate the correct patch, why add the latency of human approval?
We chose the approval gate. Not because we lacked confidence in the diagnostic accuracy, but because we talked to enough platform engineers to understand what "acting on production Kubernetes without asking" actually means in practice. This piece explains that reasoning.
The information asymmetry problem
A Kubernetes event stream tells you what is happening in the cluster right now. It does not tell you what the team knows about the cluster that is not visible in the event stream. That gap is significant, and it is where autonomous systems make their most costly mistakes.
Consider a scenario where NudgeBee correctly identifies that a deployment has CrashLoopBackOff because its memory limit is too low, and proposes raising the limit from 256Mi to 512Mi. In most cases that is the right fix. But suppose the on-call engineer knows that this particular deployment was just updated with a new release that the team suspects has a memory leak, and that raising the limit will just delay the CrashLoopBackOff by 30 minutes before the process exhausts 512Mi as well. The correct action in that context is to roll back the release, not raise the limit.
Nothing in the event stream encodes the fact that a new release was deployed 20 minutes ago and is suspected of having a memory leak. That knowledge exists in a Slack channel, in the engineer's memory, in the deploy pipeline logs. An autonomous system that applies the memory limit increase without that context does the wrong thing efficiently.
Why platform teams do not trust autonomous cluster mutation
In conversations with platform teams managing production Kubernetes clusters, the concern about autonomous remediation is not primarily about whether the AI diagnosis is correct. It is about who owns the outcome when something goes wrong.
If an autonomous system restarts a pod and that restart causes a brief service interruption at 11pm, the on-call engineer is accountable for that interruption even though they did not initiate it. The post-mortem will note that the system took autonomous action on the cluster. The engineer responsible for the cluster now has to explain a decision they did not make. That accountability mismatch is a real source of friction.
The approval gate solves this cleanly. When NudgeBee proposes a fix and an engineer approves it, the engineer has made an informed decision. They reviewed the diagnosis, applied their contextual knowledge, decided the proposed action was correct given what they know, and approved. If the fix causes a problem, the engineer can explain their reasoning. The decision was theirs. The accountability is clear.
What approval gates actually cost
The common objection to approval gates is latency. If NudgeBee identifies the root cause in 90 seconds and the engineer takes 8 minutes to approve the fix, you have added 8 minutes to your MTTR. Is that worth the accountability and trust benefits?
It depends on what the 8 minutes contain. If the engineer is asleep and takes 8 minutes to wake up, orient, and review the proposal, those 8 minutes are mostly unavoidable regardless of whether the system uses approval gates or not. The engineer needs to be awake and aware before they can make any decision, including the decision to let an autonomous system proceed.
If the engineer is already awake and at their terminal, 8 minutes to review a specific diagnosis and a specific proposed fix is a reasonable review cycle. The alternative is that the engineer spends 45 minutes investigating to reach the same diagnosis themselves. The approval gate does not add 8 minutes to a process that would otherwise take 2 minutes. It adds 8 minutes to a process that the engineer would otherwise spend 45 minutes on reaching the same endpoint.
That said, we are not claiming that approval gates are always the right answer for every class of action on every cluster. There are categories of remediation that are well-understood, low-risk, and genuinely suitable for automation without approval. Rotating expired certificates on non-critical workloads. Cleaning up completed Jobs older than a retention threshold. Scaling up an HPA replica count within pre-defined bounds during a known traffic event. For those categories, the risk of autonomous action is low and the benefit of speed is real.
The value of an approval gate is not in the seconds it adds. It is in the contextual knowledge the approving engineer brings that no event stream can capture.
Building the approval workflow that engineers actually use
An approval gate only has value if engineers engage with it rather than click through it reflexively. The worst version of an approval workflow is one where engineers approve proposals without reading them because the proposals are too verbose, the interface is too slow, or the approval is required for trivial actions where autonomous behavior would be clearly safe. That kind of rubber-stamping gives you the overhead of an approval gate without any of the benefit.
We spent considerable time on the proposal format. When NudgeBee surfaces a fix proposal in Slack or PagerDuty, the engineer sees: the identified root cause workload, the evidence from the event stream that supports that diagnosis (specific Warning events, restart counts, resource utilization), and the proposed change expressed as a diff. Not a description of the change. The actual change: which parameter, which value is changing, from what to what.
Proposed fix:
Namespace: production
Workload: payments-api
Change: memory.limit
current: 256Mi
proposed: 512Mi
Reason: 3 OOMKilled events in
last 8 min, peak RSS 241MiWith that information visible in the notification, an engineer can make a substantive review decision in under a minute. They know what NudgeBee found, they know what it wants to do, and they can apply their contextual knowledge to evaluate whether the proposed action is appropriate. If the context is that a memory-leaky release was just deployed, they can decline the proposal and initiate a rollback instead. If the context is routine traffic growth against a service with a well-understood memory profile, they can approve.
Approval gates as an audit trail
A secondary benefit of approval gates is the audit trail they produce. Every approved or declined proposal creates a record: who reviewed it, at what time, what the diagnosis was, what the proposed change was, and what the outcome was. Over time that record becomes genuinely useful.
You can see which types of incidents recur. You can see whether the same fix proposal gets approved reliably or whether engineers frequently decline it and choose a different action. A pattern of declined proposals for a specific fix type is signal that either the diagnosis is correct but the proposed fix is wrong, or the diagnosis itself is frequently incorrect for that failure mode. Either way, the audit trail makes that visible.
For teams operating under compliance requirements that mandate change management documentation, the approval audit trail also satisfies a real operational need. A record of who approved a cluster change and why is useful in a post-incident review regardless of whether compliance specifically requires it.
Where we think this goes
The right model for AI-assisted Kubernetes operations is not a binary choice between full autonomy and full manual process. It is a gradient where the appropriate level of human involvement varies by action type, risk level, and team context.
For the near term, with most teams at their current level of familiarity with AI-assisted operations tooling, requiring approval for any change that affects a running workload in production is the right default. As teams accumulate experience with the tool, as they build confidence that its diagnoses are reliable for specific failure categories, and as they develop explicit policies for which action categories they are comfortable automating, the approval requirement can be adjusted per policy rather than applying uniformly.
What we are not building toward is a fully autonomous system that applies cluster changes on its own initiative. That is not a limitation we have not gotten around to removing. It is a design principle we hold because the information asymmetry between what the event stream shows and what the team knows is not a problem that better AI solves. The team's contextual knowledge is irreducibly valuable, and the approval gate is the mechanism that puts it in the loop.