How SigNoz exposed a silent handoff failure inside a multi-agent AI workflow
My AI workflow reported that the task was complete.
The API returned 200 OK. The user interface showed every agent as completed. A final document was generated and delivered successfully.
There was only one problem.
The research agent had never returned any research.
The workflow silently caught a timeout, passed an empty response to the writing agent, and continued as though nothing had happened. The final output looked confident and polished, but it was based on incomplete context.
This was not a normal application crash. There was no obvious error page, failed request, or red status indicator.
From the outside, everything looked healthy.
SigNoz helped me see what had actually happened inside the workflow.
The Multi-Agent Workflow
The application uses several AI agents to complete one project:
- Lead Agent — understands the request and creates a plan
- Research Agent — collects relevant context and information
- Writing Agent — produces the first draft
- Review Agent — checks quality and accuracy
- Delivery Agent — prepares and exports the final result
The expected workflow is:
User Request
↓
Lead Agent
↓
Research Agent
↓
Writing Agent
↓
Review Agent
↓
Delivery Agent
Each agent depends on the result of the previous agent.
If the research agent fails, the writing agent should not continue as though valid research was available. However, my initial implementation used fallback logic that allowed the workflow to continue.
try {
researchContext = await runResearchAgent(task);
} catch (error) {
console.error("Research agent failed", error);
researchContext = [];
}
This prevented the entire request from crashing, but it created a more dangerous problem.
The system returned a technically successful response even though an essential part of the workflow had failed.
Adding Observability with OpenTelemetry and SigNoz
I instrumented the workflow using OpenTelemetry and sent the telemetry data to SigNoz.
I created one parent span for the complete project run:
project.run
Each agent execution became a child span:
lead.plan
researcher.search
writer.draft
reviewer.validate
delivery.export
I also attached attributes that describe what happened during each step.
span.setAttributes({
"agent.name": "researcher",
"task.id": taskId,
"handoff.from": "lead",
"handoff.to": "researcher",
"retry.count": retryCount,
"documents.returned": documents.length,
"fallback.used": fallbackUsed,
});
For the writing and review agents, I tracked additional values:
span.setAttributes({
"tokens.input": inputTokens,
"tokens.output": outputTokens,
"review.score": reviewScore,
"output.approved": outputApproved,
});
These attributes allowed me to investigate more than basic application uptime.
I could now see:
- Which agent was running
- How long each agent took
- Whether a handoff succeeded
- Whether fallback data was used
- How many documents were returned
- How many retries occurred
- How many tokens each agent consumed
- Whether the final output passed review
Creating the Failure
To test the workflow, I deliberately introduced a timeout into the research service.
The research agent was configured to stop waiting after [TIMEOUT VALUE] seconds.
const researchContext = await Promise.race([
runResearchAgent(task),
timeoutAfter([TIMEOUT VALUE]),
]);
The research service did not respond within the allowed time.
The application caught the exception and continued with an empty research context.
The final API response still looked successful:
{
"status": "completed",
"httpStatus": 200,
"documentGenerated": true
}
However, the trace attributes told a different story:
workflow.status: completed
researcher.documents.returned: 0
researcher.retry.count: [REAL RETRY COUNT]
fallback.used: true
review.score: [REAL REVIEW SCORE]
The workflow was technically completed but functionally degraded.
What the Trace Revealed
Inside SigNoz, the complete workflow appeared as one distributed trace.
The trace waterfall made the problem immediately visible.
project.run [REAL TOTAL TIME]
├── lead.plan [REAL TIME]
├── researcher.search [REAL TIME] — timeout
├── writer.draft [REAL TIME] — fallback context used
├── reviewer.validate [REAL TIME] — low review score
└── delivery.export [REAL TIME]
The research span consumed most of the workflow time before timing out.
The writing agent then started with no research documents, but the span itself was still marked as successful because the fallback logic prevented an exception from propagating.
This showed me an important limitation in my original monitoring logic:
A successful span did not necessarily mean that the agent completed its intended responsibility.
The system needed to track semantic success, not only technical success.
Correlating the Trace with Logs
The trace showed me where the delay occurred. The correlated logs explained why the workflow continued.
The research span contained a log similar to:
Research provider timed out after [REAL TIMEOUT] seconds.
Continuing with empty research context.
Another log from the writing agent showed:
Writing agent started with 0 research documents.
Fallback context enabled.
Without trace and log correlation, these messages would have been difficult to connect.
A timeout log by itself does not explain which user request was affected. A completed API request does not explain that an upstream agent failed.
SigNoz connected the logs to the exact trace, task, and agent span.
Building a Useful Dashboard
I created a dashboard focused on AI-agent workflow health rather than only infrastructure health.
The dashboard included:
Agent Run Duration
This panel shows how long each agent takes to complete its work.
It makes unusually slow agents and timeout patterns easier to identify.
Failed or Degraded Handoffs
This tracks handoffs where:
handoff.status != "success"
or:
fallback.used = true
Token Usage by Agent
This shows whether one agent is consuming an unusually large portion of the workflow’s token budget.
Completed Runs Using Fallback Data
This was the most important panel.
A workflow should not be considered fully successful when it completed using missing or fallback context.
Creating an Alert for Silent Failures
Traditional alerts often focus on conditions such as:
HTTP status >= 500
That would not detect this problem because the API returned 200 OK.
Instead, I created an alert for workflows that completed while using fallback data.
workflow.status = "completed"
AND fallback.used = true
I named the alert:
Successful run with degraded agent handoff
This alert captures a category of failure that normal uptime monitoring misses.
The infrastructure can be available, the API can return successfully, and the user can receive an output while the workflow is still logically incorrect.
Fixing the Workflow
The original logic treated missing research as an acceptable fallback.
if (researchFailed) {
researchContext = [];
continueWorkflow();
}
I changed it so that an essential agent failure marks the complete workflow as degraded.
if (researchFailed || researchDocuments.length === 0) {
span.setAttributes({
"workflow.status": "degraded",
"handoff.status": "failed",
"fallback.used": false,
});
throw new Error(
"Research agent failed. Drafting stopped to prevent unsupported output."
);
}
The revised behavior is now:
Research agent fails
↓
Workflow marked as degraded
↓
Writing agent does not start
↓
User receives a clear retry message
I ran the same test again.
This time, SigNoz showed that the workflow stopped at the research span instead of producing an unsupported final document.
The application no longer confused technical completion with valid completion.
What I Learned
Before adding observability, I could see whether the application returned a response.
After adding SigNoz, I could see whether each agent actually fulfilled its responsibility.
That distinction matters in AI systems.
A normal application often fails through an exception, unavailable service, or unsuccessful HTTP request.
An AI workflow can fail more quietly:
- An agent returns empty context
- A retry produces duplicate work
- A fallback hides an upstream failure
- A reviewer approves an incomplete response
- The workflow completes while violating its own assumptions
The most dangerous AI failures are not always crashes.
Sometimes every technical component appears healthy while the final result is still wrong.
SigNoz helped me observe the complete chain of decisions, handoffs, retries, logs, and outputs behind one AI-generated result.
The dashboard originally told me that my AI team had completed the task.
The trace showed me the truth: one of the most important agents had never completed its work.
