Observability at Capital One
- What
- Enterprise platform observability for 5,000+ internal applications — the OpenTelemetry ingestion pipeline, the monitoring estate, and a multi-agent AI incident-analysis platform, led as tech lead across three teams (18 engineers).
- Why it was hard
- The volume is 100–300 terabytes of telemetry a day, the consumers are on-call engineers who need it in seconds not hours, and the legacy estate was ~500,000 hand-maintained monitors accreted over years.
- What I decided
- Land everything in one queryable lake (OTel → transform → Apache Iceberg, sub-20s end-to-end); rebuild the monitor estate as ~150 parameterized templates instead of migrating half a million monitors 1:1; and put agents — not more dashboards — between the telemetry and the humans.
- What happened
- Incident resolution went from 96 minutes to 36. Monitor maintenance overhead dropped ~95%. The pipeline holds sub-20-second end-to-end at 100–300 TB/day.
Closed source. Figures are from internal measurement, as reported on my CV — this page can show the shape of the work, not the code.
One lake instead of many dashboards
The pipeline collects OpenTelemetry data from 5,000+ applications, transforms it in stream, and writes to an Apache Iceberg datalake with sub-20-second end-to-end latency at 100–300 TB/day. The unglamorous decision that everything else depends on: telemetry lands in one queryable place with real table semantics, rather than fanning out into per-team tools. Everything downstream — the monitors, the agents, the cost model — is a view over that lake.
Deleting 499,850 monitors
The migration from New Relic to Observe Inc could have been a lift-and-shift of ~500,000 monitors. Most of them were near-copies of each other, differing only in an application name and a threshold — which is to say, they were data pretending to be configuration. We rebuilt the estate as ~150 parameterized monitors that expand over the application inventory, cutting maintenance overhead by roughly 95%. The judgment call was refusing the “faithful” migration: porting the mess would have preserved half a million opportunities for drift, and the deadline pressure to just copy them was real.
Agents between the telemetry and the humans
On top of the lake sits a multi-agent incident-analysis platform — LangGraph, ReAct-style reasoning, tool-calling against the telemetry sources, with memory and evaluation loops. Median incident resolution went from 96 minutes to 36. Two design points carried most of that:
- Shared context, computed once. Incident snapshots are cached and shared across concurrent users, so ten engineers staring at the same outage don’t pay for ten identical LLM analyses. Three caching layers sit in the inference path; this was the subject of my talk “Taming the LLM Bottleneck” at Async Live 2026, Capital One’s national engineering conference.
- Agentic context building as a first-class interface. A genAI pilot pod I led built an Observability MCP server, so any agent — ours or a team’s own — can assemble incident context from the lake through one protocol instead of bespoke queries.
Inference runs on self-hosted open-weight models — Fargate services on Kubernetes — sized for hundreds of parallel analysis streams.
Also in this role: a serverless gateway architecture backing CI/CD for the same 5,000+ applications, first place at Capital One’s national GenAI hackathon (funded to production), and the AI second-brain initiative across a 1,700-person vertical.