There is a predictable cliff in observability architecture that happens somewhere between clusters five and fifteen. Before that point, Prometheus scraping everything into a single remote-write target works. Grafana dashboards load fast enough. Alert rules referencing cluster="prod" are unambiguous. After that point, the same setup generates gigabytes of label sets per day, alert managers route to the wrong channel half the time, and the on-call engineer opens a Grafana dashboard to find a sea of overlapping time series where none of the panel titles mention which cluster is actually on fire.
This is not a criticism of Prometheus or Grafana. These tools are excellent. The problem is that a single-cluster mental model gets baked into the observability stack at deployment time, and nobody revisits it until the pain is significant enough to justify a week of infrastructure rework.
Here is what the architecture decisions actually look like when teams cross the ten-cluster mark.
Cardinality Is the First Thing That Breaks
When you have one cluster, you label your metrics with namespace, pod, container, and maybe a team label. At one cluster, a few hundred pods, this produces a few hundred thousand time series. That is fine.
At fifteen clusters, multiply everything by fifteen. But the real multiplier is not the cluster count alone. It is that each new cluster typically has its own set of namespaces, which have their own pods, which have their own containers, which have unique names. The label set explodes combinatorially. Prometheus TSDB is designed to handle this, but your remote-write pipeline, your Thanos or Cortex ingester, and especially your query path are not always sized for the resulting cardinality.
The first architectural decision worth making early: add a cluster label with a short, stable identifier at the metrics source, not at the aggregation layer. Doing it at the aggregation layer via relabeling rules is tempting because it feels centralized, but it means your retention policies and recording rules all have to account for unlabeled pre-ingestion data. Doing it at the source, typically via external_labels in your Prometheus configuration, means every metric arrives with its cluster identity intact, and you can filter or aggregate from the start.
The Federation vs Centralization Decision
When teams grow past ten clusters, the instinct is often to push everything to one central Prometheus or one Thanos receive. This works, but it creates a single point of both failure and cost. A better framing is to ask what questions you actually need to answer at the global level versus what questions are cluster-local.
Cluster-local questions: is this pod CrashLoopBackOff? Is this node under memory pressure? Is this HPA unable to scale because no healthy nodes have capacity? These are best answered from a Prometheus instance co-located with the cluster. The latency is low, the scrape targets are local, and a cluster failure does not take down your ability to observe other clusters.
Cross-cluster questions: which clusters have elevated error rates right now? How does the p99 latency of service X compare across staging and production regions? These require a global view. Thanos query frontend with sidecar-mode prometheus instances is one common pattern. Grafana Mimir with separate ruler storage is another. The point is that you only lift metrics to the global layer that actually require global context.
A third approach that works well for teams with stable cluster sets: recording rules at the cluster level that produce pre-aggregated metrics, which are then federated up. Instead of shipping raw container_cpu_usage_seconds_total with all its labels for every pod, you ship a cluster:cpu_used_cores:sum recording rule result. The global layer has enough resolution for cross-cluster comparison without carrying the full cardinality.
The teams that survive the ten-cluster transition without a full rewrite are almost always the ones that decided early which questions need a global answer and which ones are purely local.
Log Aggregation: Avoid the Single Giant Pipeline
Centralizing logs is even more tempting than centralizing metrics, because logs are where you go when metrics fail to explain the incident. But a single Loki or Elasticsearch cluster receiving logs from fifteen production clusters is a single point of failure that can make an incident actively worse. If your central log aggregation is overwhelmed exactly when a high-cardinality failure is flooding it with stack traces, you have lost visibility at the worst possible time.
The patterns that hold up under pressure:
- Regional aggregation before global. If you have clusters in three AWS regions, aggregate per-region first, then federate to a global view. Each regional aggregator can absorb its local traffic without any single aggregator taking the full combined load.
- Separate ingestion from query. Using a log shipper like Fluent Bit or Vector to push to S3 or GCS with a query layer like Athena or BigQuery on top gives you durability without a hot path single point of failure. The tradeoff is query latency, which is fine for post-incident review but poor for live triage.
- Structured logging enforced at the cluster admission level. This is underrated. The cost of parsing unstructured logs at query time across fifteen clusters is enormous. A ValidatingWebhookConfiguration that rejects pods whose logging configuration does not produce JSON output pays for itself within a few months of log volume.
Alert Routing Across Clusters
At one cluster, all alerts go to one Slack channel and maybe one PagerDuty escalation policy. This is easy. At fifteen clusters, you have fifteen teams (or sub-teams), each owning different namespaces within different clusters, and your Alertmanager routing tree becomes a maze of matchers that no single person fully understands.
The routing problem has two failure modes. The first is under-routing: alerts that should go to a specific team land in a general channel, sit for twenty minutes, and finally get someone's attention only after a user-facing impact is visible. The second is over-routing: every alert goes to every channel, alert fatigue sets in, and the channel becomes background noise.
What works: structuring alert labels with enough context that routing rules can be simple and static. An alert with labels cluster="prod-eu-1", team="payments", severity="critical" can be routed unambiguously by a simple Alertmanager receiver tree. The work is not in the routing tree itself but in ensuring that every alerting rule in every cluster produces these labels consistently.
One specific decision that helps: separate Alertmanager instances per cluster, not one shared instance. Shared Alertmanager is tempting for deduplication, but deduplication across clusters is rarely what you want. A CrashLoopBackOff in the payments namespace in prod-eu-1 is not the same alert as the same pod name CrashLoopBackOff in prod-us-1, even though they might deduplicate to the same fingerprint in a naive setup. Per-cluster Alertmanager instances with a shared inhibition config (for true global inhibitions like maintenance windows) gives you both isolation and cross-cluster coordination where it actually matters.
Distributed Tracing When Services Cross Cluster Boundaries
If you run a microservices architecture where some services call services in other clusters, trace context propagation becomes mandatory for meaningful root cause analysis. A trace that loses its parent span ID at the cluster boundary produces two orphaned traces, and correlating them manually during an incident takes time you do not have.
W3C trace context propagation headers are now standard enough that most service meshes (Istio, Linkerd) inject them by default. The operational piece that often falls through: ensuring that every service running in every cluster has its tracing SDK configured to connect to the cluster-local OTLP collector, not a hardcoded external endpoint. Hardcoded endpoints create cross-cluster network dependencies on the hot path, which means a network partition between clusters can generate tracing overhead that exacerbates the very incident you are trying to observe.
For trace sampling at scale, a head-based sampling rate of 10-20% at the cluster level is typical for high-volume services. More useful for production debugging is tail-based sampling: record 100% of traces that contain an error span, regardless of the sampling decision made at trace initiation. This gives you complete traces for all failures without the storage cost of recording everything.
What to Stop Doing Past Ten Clusters
Not all single-cluster patterns are worth adapting to multi-cluster. Some should be abandoned.
The "check the dashboard first" incident response flow breaks badly at scale. A dashboard that spans fifteen clusters with meaningful drill-down takes too long to build and too long to interpret under pressure. The teams that handle multi-cluster incidents well do not open dashboards first. They start from the alert label context, filter to the affected cluster and namespace, and let the tools do the initial scoping. Dashboards are for pattern analysis after the fact, not for the first five minutes of incident response.
Similarly, manual runbook-per-alert maintenance does not scale. A team maintaining fifteen clusters with fifty alerting rules each has 750 potential runbooks to keep current. In practice, most of those runbooks go stale within a quarter. The scalable version is parameterized runbooks that reference cluster and namespace from the alert context, rather than fifteen separate documents describing the same OOMKilled recovery procedure for each cluster.
Finally, the expectation that one person can understand the full observability stack end-to-end does not hold past a certain scale. The architect who designed the original Prometheus deployment often becomes the bottleneck for every configuration change. Building observability configuration as code, with PR review by multiple team members and tested against a staging cluster before rollout, removes this bottleneck. It also creates an audit trail of why certain alert thresholds are set the way they are, which matters more than it sounds when you are debugging why an alert is or is not firing in an incident.
The Honest Counterpoint
All of these patterns add operational complexity. Federated Prometheus with Thanos sidecars requires you to understand Thanos. Per-cluster Alertmanager instances require you to keep configurations synchronized. Structured logging enforcement via admission webhooks requires you to maintain the webhook itself.
The alternative, a simpler setup that still works well enough, is often the right call for teams that are at eight or nine clusters and not growing fast. The architecture decisions described here pay off when you are either growing quickly past ten clusters or when your existing setup is already producing alert fatigue or observability blind spots. If neither of those is true, delaying the investment is rational.
What is not rational is waiting until you have twenty-five clusters and a major incident before addressing the architecture. The observability cliff tends to produce its worst effects at the worst possible time: during a multi-cluster incident, when the tools that should help you triage are themselves overwhelmed by the event volume they are supposed to be tracking.