Before the model said a word, an LLM client told the endpoint all of this about my machine:
x-stainless-arch: arm64
x-stainless-async: false
x-stainless-lang: python
x-stainless-os: MacOS
x-stainless-package-version: 3.35.1
x-stainless-read-timeout: 600
x-stainless-retry-count: 0
x-stainless-runtime: CPython
x-stainless-runtime-version: 3.14.7
That block is not from a document. It is from a capture — I put a logging proxy between a real OpenAI client and an endpoint, made one call, and read the wire. Two other things were in the capture: the client silently retried after a 500, resending the entire conversation including the system prompt, and stamped the retry with a header counter. The proxy saw two requests. The application saw one call. Nobody's dashboard shows you that — because the dashboards belong to the party on the other side of the wire.
The rant is right, and the thread proves the demand
Yesterday a post by Jared Siegel made the front page of Hacker News with a title that is its own executive summary: If you don't run your own LLM proxy, how the fuck do you even know whats going on?. The post is a rant with receipts of a different kind — screenshots of his own proxy dashboards — and its sharpest lines are about what the provider's analytics actually answer: "I can't make a decision with this bullshit." About the pretty usage graphs: "That is a distraction. That is not token usage statistics with time series data meant for engineers." His prescription is one sentence: "I'm not going to suggest a proxy provider for you to use, only that you run it yourself somewhere you control."
The discussion under it is the demand in the wild. One commenter built their own agent harness "to have actual full observability of what it is doing and why it is doing what it is doing." Another asked the question the whole thread orbits: "Seeing some good stuff in the comments, but how are people doing this?" The practical answers name real tools — LiteLLM for virtual keys, so "provider API keys never go near the sandbox" — and the skeptical ones land on the same worry from the other side: the proxies themselves are young, and "with the velocity of updates I feel there has to be vulnerabilities there."
What nobody in the thread posted was the smallest thing that answers the question: a capture. Not a dashboard, not a product, not a config guide — the actual bytes. So here is one, runnable on your machine in two commands, no API key, and what it caught surprised even me.
The capture: one file, two servers, zero keys
The setup is one Python file. Two servers: a mock provider that speaks just enough OpenAI to answer a chat completion (and fails the first attempt on purpose — you will see why), and a logging reverse proxy that forwards every byte and keeps a verbatim copy. The client is the real OpenAI SDK — the same package your scripts and agents import — pointed at the proxy instead of the internet.
#!/usr/bin/env python3
"""proxydemo.py — see what actually leaves your machine when you call a model.
Two servers, one file, zero API keys:
:8081 a mock OpenAI-compatible provider (it fails the first attempt on purpose)
:8080 a logging reverse proxy that tees every byte to stdout
The client is the real OpenAI Python SDK, pointed at the proxy.
Run it:
pip install openai
python3 proxydemo.py
"""
import json
import threading
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
CAPTURE = []
class MockProvider(BaseHTTPRequestHandler):
"""Pretends to be api.example.com. The first request fails with a 500."""
attempts = 0
def do_POST(self):
self.rfile.read(int(self.headers.get("content-length", 0)))
MockProvider.attempts += 1
if MockProvider.attempts == 1:
self.send_response(500)
self.send_header("content-length", "0")
self.end_headers()
return
out = json.dumps({
"id": "cmpl-mock", "object": "chat.completion", "model": "mock-1",
"choices": [{"index": 0,
"message": {"role": "assistant", "content": "MOCK REPLY"},
"finish_reason": "stop"}],
"usage": {"prompt_tokens": 9, "completion_tokens": 2, "total_tokens": 11},
}).encode()
self.send_response(200)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(out)))
self.end_headers()
self.wfile.write(out)
def log_message(self, *a):
pass
class LoggingProxy(BaseHTTPRequestHandler):
"""Forwards to the mock and keeps a verbatim copy of everything."""
def do_POST(self):
body = self.rfile.read(int(self.headers.get("content-length", 0)))
CAPTURE.append({"kind": "request", "method": self.command, "path": self.path,
"headers": dict(self.headers), "body": body.decode()})
fwd = {k: v for k, v in self.headers.items()
if k.lower() not in ("host", "content-length", "connection")}
req = urllib.request.Request("http://127.0.0.1:8081" + self.path,
data=body, headers=fwd, method="POST")
try:
with urllib.request.urlopen(req) as r:
status, rbody, ctype = r.status, r.read(), r.headers.get("content-type", "application/json")
except urllib.error.HTTPError as e: # pass provider errors through, untouched
status, rbody, ctype = e.code, e.read(), e.headers.get("content-type", "text/plain")
CAPTURE.append({"kind": "response", "status": status, "body": rbody.decode()})
self.send_response(status)
self.send_header("content-type", ctype)
self.send_header("content-length", str(len(rbody)))
self.end_headers()
self.wfile.write(rbody)
def log_message(self, *a):
pass
def main():
for srv, port in ((MockProvider, 8081), (LoggingProxy, 8080)):
httpd = ThreadingHTTPServer(("127.0.0.1", port), srv)
threading.Thread(target=httpd.serve_forever, daemon=True).start()
from openai import OpenAI # the real client, pointed at the proxy
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="sk-demo-not-a-real-key")
reply = client.chat.completions.create(
model="mock-1",
messages=[{"role": "system", "content": "You are a test harness."},
{"role": "user", "content": "Reply with: MOCK REPLY"}])
print("client received:", reply.choices[0].message.content, "\n")
n = 0
for e in CAPTURE:
if e["kind"] == "request":
n += 1
print(f"=================== request #{n} -> provider ({e['method']} {e['path']}) ===================")
for k in sorted(e["headers"]):
print(f" {k}: {e['headers'][k]}")
print(f" body: {json.dumps(json.loads(e['body']))[:220]}")
else:
print(f"------------------- provider answered: {e['status']} -------------------\n")
print(f"the proxy saw {n} requests. your application saw one call.")
if __name__ == "__main__":
main()
I ran it. The output below is verbatim from that run, trimmed only where the JSON repeats:
client received: MOCK REPLY
=================== request #1 -> provider (POST /v1/chat/completions) ===================
Accept-Encoding: gzip, deflate, zstd
Connection: keep-alive
Content-Length: 136
Host: 127.0.0.1:8080
accept: application/json
authorization: Bearer sk-demo-not-a-real-key
content-type: application/json
user-agent: OpenAI/Python 3.35.1
x-stainless-arch: arm64
x-stainless-async: false
x-stainless-lang: python
x-stainless-os: MacOS
x-stainless-package-version: 3.35.1
x-stainless-read-timeout: 600
x-stainless-retry-count: 0
x-stainless-runtime: CPython
x-stainless-runtime-version: 3.14.7
body: {"model": "mock-1", "messages": [{"role": "system", "content": "You are a test harness."}, {"role": "user", "content": "Reply with: MOCK REPLY"}]}
------------------- provider answered: 500 -------------------
=================== request #2 -> provider (POST /v1/chat/completions) ===================
x-stainless-arch: arm64
x-stainless-os: MacOS
x-stainless-package-version: 3.35.1
x-stainless-read-timeout: 600
x-stainless-retry-count: 1
x-stainless-runtime-version: 3.14.7
body: {"model": "mock-1", "messages": [{"role": "system", "content": "You are a test harness."}, {"role": "user", "content": "Reply with: MOCK REPLY"}]}
------------------- provider answered: 200 -------------------
the proxy saw 2 requests. your application saw one call.
I ran it twice and diffed: the captures are byte-identical. This is not an anecdote, it is an instrument — anyone with Python can produce the same two requests on their own machine in under a minute.
What the capture actually shows
The machine inventory, volunteered on every call. The x-stainless-* block is the SDK's telemetry: your operating system, your CPU architecture, your runtime and its exact version, the package version, whether the call was async, the read timeout. It is not hidden — it is on the wire, which is different: it is invisible to anyone who does not capture. Your client is a sensor reporting on your environment, and the sensor's dashboard is on their side, not yours.
The silent retry. The mock fails the first attempt with a 500, and the SDK does what SDKs do: it retried, resending the identical conversation — system prompt included — and stamped the second attempt with x-stainless-retry-count: 1. Your application called the API once. The conversation crossed the wire twice. On the provider's side this can look like one successful call and some noise; on your side, without a capture, it looks like nothing at all. Every "why did my token bill double" question starts here.
Everything is readable on the path. The system prompt you consider private, the whole conversation, and the bearer token all travel as plaintext JSON between your process and the endpoint — through every proxy, gateway, and middleware that sits between them anyway. The question is never whether intermediaries can read it. It is whether the one who keeps the log is you.
The rung this sits on
There is a pattern underneath, and the authority ladder this series has been building names it: evidence you merely get to see is a caller label; evidence that lives on infrastructure you control is the floor. The provider's usage dashboard is their receipt about you. Jared's one-line prescription — run it yourself somewhere you control — is the same move as pinning a registrar or keeping your own journal: the log moves to the side of the party who needs to audit. A proxy you run is not a better dashboard. It is your dashboard.
What to steal
- The provider's dashboard answers their questions, not yours. Token graphs are marketing-shaped. A capture answers engineering questions: what left, how many times, with what attached.
- Virtual keys are the grown-up version of this demo. LiteLLM (61k+ stars) sits between your code and every provider, issues per-agent keys, and keeps the real ones out of your sandboxes — the thread's most repeated piece of advice, and it is the same proxy position with more features.
- Any control that touches your traffic should log, and the log should be yours. A gate that emits to someone else's analytics is a control you cannot audit — the bypass-without-a-log problem from yesterday's Sphinx debate, one layer down.
Two honest limits
The provider side of this demo is a mock, because there is no API key on this machine. Everything the client emits — headers, retry, plaintext conversation — is real and captured verbatim; the model's answer is canned. The mock is also the point: the capture works identically against the real endpoint, and the two commands below let anyone with a key produce the real version in minutes.
A 90-line stdlib proxy is an instrument, not infrastructure. It speaks plain HTTP; real agents speak TLS, and for those the tool is mitmproxy (45k+ stars) with your own CA — same receipts, one config file away. I did not run a frontier coding agent against this capture; the reader who does will find more headers than I did, and that finding belongs in the comments.
Your turn
Run the two commands — pip install openai, save the script, python3 proxydemo.py — and read your own capture. Then point a real agent at the same proxy (OPENAI_BASE_URL is one environment variable on most tools) and count three things: how many requests one user action produces, what headers your client volunteers about your machine, and how many retries your application never mentioned. Post the three numbers. That is the state of your agent's traffic, measured instead of assumed.
