Getting a Hermes agent running takes about thirty minutes.
Getting real observability on it takes considerably longer. You pick a backend. You stand up an OTel collector. You configure the exporter. You wire env vars in every place the agent runs. Then you work out which spans your agent emits on its own and which ones you have to instrument by hand, deploy, and wait to find out whether anything actually arrives.
That gap is not a tooling gap. The tools are good. The gap exists because the model underneath distributed tracing was designed for a different shape of program than the one you are running.
The model mismatch
Distributed tracing assumes a request tree. A call comes in, it fans out to a known set of downstream services, each hop is a span, and the tree closes. The structure is known before the request starts. Your job as an instrumenter is to label edges that already exist.
An agent is not a request tree. It is a loop with a model in the middle deciding what happens next. The structure of a run is not known before the run starts, and it is not stable between runs. Four things follow from that, and each one costs you a day.
The interesting unit is the tool call, not the network hop. When an agent run goes wrong, the question is almost never which service was slow. It is what did it decide to do, with what arguments, and what came back. Those are semantic events. HTTP instrumentation captures none of it for free, so you end up hand instrumenting the things you actually care about, which is the expensive kind of instrumentation.
The same input produces different traces. Two runs of the same task can take different paths, call a different number of tools, and both succeed. Aggregate span statistics assume a stable topology, so p95 latency on a span that appears three times in one run and zero times in the next is a number that means very little. You want to compare runs, not aggregate spans, and most backends are not organized around that.
Most agent failures are not errors. This is the one that surprises people. A tool call fails, the agent reads the error, decides it can work around it, and carries on. The run completes. The final message says the task is done. Every span is green and your error rate is zero, and the work is wrong. There is nothing for a threshold alert to fire on, because at the transport layer nothing went wrong. The failure is in the reasoning, two steps upstream of anything you are measuring.
The payloads are the point, and they are large. Prompts, tool arguments, and tool results are what you need to see. Span attributes were not designed to carry multi kilobyte text blobs, and once you start truncating them you have thrown away the evidence you were collecting in the first place.
OpenTelemetry is working on GenAI semantic conventions and they will help. But conventions standardize field names. They do not change the fact that you are asking a request tree model to describe a decision loop, and they do not shorten the afternoon you spend wiring a collector before you can look at your first run.
What we did instead
We built Failproof around the tool call boundary rather than the network boundary, because that is where agent behavior is legible. Every action the agent takes is an event: the call, the arguments, the result, and where it sat in the run. Installing it is a package and a command.
npm i -g failproofai
failproofai config
config sets up the binaries and the environment so that your Hermes logs get forwarded to your dashboard. Pick the recommended option when it prompts you, and it will ask for an API key.
To get one, open your dashboard, go to the Keys section, and create a new key with the events.add permission. That is the permission that lets the CLI send log events. Name it after the agent it belongs to, which makes life easier later when you have several.
Paste the key into the CLI. That is the whole setup. No collector to run, no exporter to configure, and no changes to your agent code.
Events start arriving in the dashboard as the agent works. If you have been running the agent already, you can pull in history you did not capture at the time:
failproofai backfill --since 7d
What to look at in your first hour
Once data is flowing, the useful move is not to build dashboards. It is to open three or four real runs and read them end to end. Three things tend to show up on a first pass.
Repeated calls. The same file read three times in one run, or the same lookup fired on every iteration of a loop. Usually harmless for correctness and expensive for latency and tokens. Easy to fix once you can see it, invisible until then.
Swallowed errors. Find a run that succeeded and scan for failed tool calls inside it. Where the agent hit an error and continued, ask whether continuing was actually right. This is where the interesting bugs are.
Divergence. Run the same task twice and put the two traces side by side. Where they differ is where your agent is making a judgment call you never specified. Sometimes that is fine. Sometimes it is a prompt that needs to be tighter.
None of these produce an exception. None of them page you. They are the difference between an agent you deployed and an agent you understand, and the only way to find them is to look at what actually happened.
The CLI is open source. Docs and dashboard at befailproof.ai.
