A dependency graph built from your cluster's manifests answers a precise question: what did you declare? Every edge — a Deployment that mounts a Secret, a Service that routes to a Pod, a RoleBinding that grants access — is something written down in the API server. It's the map of the system as designed.
But the map as designed and the system as run are not the same thing. A Service can front five workloads and, in practice, only ever get called by one. A "deprecated" internal API can still be taking live traffic from a service nobody remembers wiring up. The declared graph can't see any of this, because a service-to-service call at runtime leaves no declarative trace — it's just packets. This guide is about closing that gap: overlaying what your workloads actually do onto the graph of what they were declared to do, using the OpenTelemetry runtime overlay introduced in KubeAtlas v1.5.
Why the declared graph has a blind spot
Kubernetes models intent, not behaviour. The objects in etcd describe relationships you configured; they say nothing about which of those relationships carry traffic, or about calls that happen entirely outside the object model. Two failure modes hide in that blind spot:
- Declared-but-dead edges. A Service selector still matches an old Deployment, a NetworkPolicy still allows a path — but no request has traversed it in weeks. It looks load-bearing on the graph and is safe to remove; you just can't tell which is which.
- Observed-but-undeclared calls. Service A calls Service B directly by cluster DNS, with nothing in either manifest to record the relationship. It won't appear on a declarative graph at all — until the day B has an incident and you're reverse-engineering who depends on it from logs.
The uncomfortable part: the edges you most want to trust before a change are exactly the ones the manifest can't confirm. "Is this Service still used?" and "who actually calls this thing?" are runtime questions, and they need a runtime signal.
The runtime signal you (probably) already emit
That signal is distributed tracing. If your services are instrumented with OpenTelemetry — directly, or through an auto-instrumentation agent or a service mesh — every request already carries the caller, the callee, and the timing. A trace is an observed edge; a stream of traces is the observed call graph. The problem has only ever been that traces live in Jaeger or Tempo, and your dependency graph lives somewhere else, and nothing puts them side by side.
KubeAtlas v1.5 puts them side by side, without merging them. The design rule that makes this safe is worth stating up front: observed edges never enter the declarative graph. They ride on top as a separate, clearly-typed overlay you can toggle on and off. The map of intent stays clean; the map of behaviour is a layer you add when you want it.
How KubeAtlas models it: the OTel overlay
The overlay has three parts, all of them opt-in and off by default:
- A standalone OTLP/gRPC receiver on
:4317that accepts trace spans. It drops on backpressure rather than blocking your senders, and when the overlay is disabled it isn't even listening — zero overhead on the core graph. - A background correlator that reads the persisted spans and infers
CALLS_AT_RUNTIMEedges between workloads — a distinct, overlay-only edge type that is never written into the declarative graph. - A comparison mode that classifies each edge as declared-and-observed, declared-but-unobserved, or observed-but-undeclared — which is the whole point.
Good to know: the receiver is a Tier 2 feature — it persists spans, so it's meant for an instance with storage, not the ephemeral in-memory install. Nothing about it changes the declarative graph or the /api/v1 responses you already consume; the overlay is strictly additive.
Step 1 — Enable the overlay
The runtime overlay ships in v1.5.0. Enable the receiver on a persistent instance and point your OpenTelemetry exporters (or your collector's OTLP exporter) at it:
helm upgrade --install kubeatlas oci://ghcr.io/lithastra/charts/kubeatlas \
--version 1.5.0 \
--namespace kubeatlas --create-namespace \
--set otel.overlay.enabled=true
kubectl -n kubeatlas rollout status deploy/kubeatlasThen send spans to the receiver's OTLP/gRPC endpoint on port 4317 — the same protocol your collector already speaks, so in most setups this is one extra exporter, not a re-instrumentation project. Exact chart keys and collector snippets are in the documentation.
Step 2 — Toggle it on in the web UI
Open the KubeAtlas web UI and flip the "Show OTel overlay" toggle. The declared graph you already know renders as before; the observed CALLS_AT_RUNTIME edges are drawn on top in their own style, so the two layers are visually distinct. With the overlay on you can:
- See which declared paths actually carry traffic — and which are dark.
- See runtime calls that have no declared edge underneath them, surfaced as undeclared dependencies.
- Select a workload and read its real callers and callees, not just its configured ones.
Step 3 — Query the overlay from the API
The same data is available through two endpoints, which is what you'd wire into a review gate or a periodic report:
# the observed runtime call edges
curl -s localhost:8080/api/v1/otel/overlay | jq '.edges[]'
# declared-vs-observed classification for every edge
curl -s 'localhost:8080/api/v1/otel/overlay?compare=true' \
| jq '.edges[] | {from, to, status}'
# the underlying spans behind an edge
curl -s localhost:8080/api/v1/otel/traces | jq '.[].service'The ?compare=true response is the one that pays off: each edge comes back tagged as declared-and-observed, declared-but-unobserved, or observed-but-undeclared. That single field turns two vague worries — "is this still used?" and "what did we forget to write down?" — into a filterable list you can act on.
Step 4 — In Headlamp and Backstage
The overlay isn't confined to the KubeAtlas UI. The Headlamp plugin (v1.2.0) adds an OTel Overlay view with a TraceTimeline that deep-links straight to the underlying trace in Jaeger or Tempo — so you go from "these two services talk" to the actual trace in one click, without leaving your cluster dashboard. The Backstage plugin reaches GA at v1.0.0 and gains a Runtime calls (OTel) card alongside its Admission policies card, putting observed dependencies on the service catalog page where an on-call engineer already looks.
Operability: the receiver exports its own Prometheus metrics — kubeatlas_otel_received_total, kubeatlas_otel_dropped_total, kubeatlas_otel_written_total, kubeatlas_otel_unmatched_spans_total, and kubeatlas_otel_runtime_edges_total among them — so you can see exactly how many spans arrived, how many were dropped under backpressure, and how many runtime edges the correlator has inferred.
The owl's-eye summary
| Question | Declared graph alone | KubeAtlas v1.5 + OTel overlay |
|---|---|---|
| What was this system designed to do? | The whole graph | The base layer, unchanged |
| Which declared edges carry real traffic? | Invisible | Declared-and-observed edges |
| Is this Service still actually called? | Guess from selectors | Declared-but-unobserved flag |
| Who calls this thing that we never wired up? | Not on the graph | Observed-but-undeclared edges |
| Show me the trace behind an edge | Separate tool | TraceTimeline → Jaeger/Tempo |
A declarative graph tells you the truth about your intentions. Traces tell you the truth about your traffic. Kept apart, each has a blind spot the other fills; laid one over the other, "what did we build, what does it actually do, and where do those disagree?" becomes a question you can answer on one canvas — with the map of intent still intact underneath.