Uncategorized
If your OpenAI traffic runs through LiteLLM, Portkey, Kong AI Gateway, or an internal gateway, your cost attribution model is already broken.
Every upstream caller collapses into a single workload identity at the exact point where money starts being spent. Provider billing, tag-based FinOps tools, and cloud cost platforms cannot split that cost back by service, team, feature, or customer.
This is the LLM gateway blind spot. It’s is quickly becoming a core problem in DevoOps and FinOps for AI, where token-based spend and shared infrastructure break traditional cost attribution models.
It is the fastest-growing gap in cloud cost attribution, and it follows the same pattern teams have seen with Kafka, Snowflake, and other shared systems. The difference is that now the spend is token-based, growing fast, and directly tied to product usage.
Why the chain breaks at the gateway
A gateway-mediated call has two hops:
upstream-svc ──► llm-gateway ──► api.openai.com
The identity you care about lives on the first hop: service, tenant, feature, customer.
The billable event happens on the second hop: token usage.
Provider invoices only see the gateway identity. Once that collapse happens, downstream systems cannot reconstruct it.
Every standard data source sits after the collapse:
- Provider billing exports show cost per API key or project, not per service
- Cloud billing (CUR, GCP, Azure) doesn’t see OpenAI spend or still attributes it to the gateway
- Tag-based FinOps tools assign cost to the resource that spent it: llm-gateway
The result is a shared-infrastructure cost bucket instead of usable unit economics.
What this looks like in practice
| View | Default (collapsed) | What the business needs |
| Cost by workload | llm-gateway = $X | Split across real services by token usage |
| Cost by customer | Shared infra | Per-customer AI cost |
| Cost by feature | AI looks free | Real AI COGS per feature |
| Pricing signal | Aggregate % of cost | Customer-level profitability |
This is not just a visibility problem. It directly impacts:
- Customer profitability analysis
- Pricing decisions
- Feature ROI
- Cloud cost optimization
Why tagging and logs fail
Most teams try to fix this with header propagation and logs. It works in staging. It fails in production.
1. Propagation drifts
New services, retries, or alternate paths create missing attribution. Over time, a growing share of spend falls into “unknown.”
2. Logs are not a cost system
Sampling, retention limits, and ingestion bias break the dataset. Joining logs to billing becomes a fragile pipeline.
3. Shared calls cannot be split with tags
Retries, batching, caching, and fan-out workflows mean multiple callers share one provider request. Tags cannot fairly allocate cost. Only token-level accounting can.
4. Logs miss negative space
Failed calls, retries, and fallbacks still create infrastructure cost pressure but are not fully captured.
This is why teams searching for ways to allocate shared cloud costs across microservices without tagging get stuck. Tagging narrows the gap. It does not close it.
Runtime attribution: observe the request, don’t reconstruct it
The only reliable solution is to observe cost at runtime instead of reconstructing it from billing data.
Attribute uses an eBPF sensor that runs alongside the gateway and observes both sides of the request:
- First hop: upstream service → gateway (identity intact)
- Second hop: gateway → provider (token usage and cost)
By correlating both, it maps provider cost back to the originating workload based on actual token consumption.
This enables:
- Kubernetes cost visibility by source workload
- Customer-level AI cost attribution
- Feature-level AI COGS
- Real-time cost signals
- Accurate cloud cost allocation without tagging
Implementation details that matter
- Read-only sensor
- PII and secrets stripped before any data leaves the environment
- Minimal OpenTelemetry events over TLS with JWT authentication
- Works with existing OTel collectors
- No sidecars, no code changes, no gateway config changes
The result: the llm-gateway cost bucket disappears. Every OpenAI dollar is tied to a workload, team, feature, and customer.
Evaluation checklist
If you are evaluating FinOps tools for AI and shared infrastructure, ask:
- Can it split gateway cost by originating workload without tag propagation?
- Can it attribute AI cost to customers and features?
- Does it read live traffic or billing exports?
- Can it split shared calls proportionally by token usage?
- Can telemetry stay inside your environment?
Most tools fail on multiple points because they start after the billing event.
LLM gateways break cost attribution at the exact point where spend begins. If your FinOps model starts with billing data, the most important signal is already gone. The only way to recover it is to observe the system while it runs. That is the difference between estimating cost and actually understanding it.
FAQ
Why can’t tagging solve this?
Because shared calls and system drift break attribution. Tags do not capture proportional usage.
Why not use logs?
Logs are incomplete, sampled, and not designed for cost allocation.
Is attribution based on requests or tokens?
Tokens. Request-based allocation produces incorrect cost models.
What about security?
No raw packets leave the environment. Data is stripped, minimized, and sent via secure telemetry.
