There's Now an Open-Source Fault Localization Project in Ops

Ops rarely lacks metrics, topology, traces, or logs. What it lacks is lining those four up when something breaks.

Take an entry service that returns 200 with a 0% error rate. It looks fine, so the investigation often stops on that page. The real problem is downstream — SQL already threw — and this page never shows it.

Classic monitoring · four pages, each does one job, a human joins them

• Metric — who is slow, who is erroring
• Topology — which upstream/downstream node went red
• Trace — which hop ate the time
• Log — search the original line by call ID

You line up the time window yourself, copy the call ID yourself, search again on the next page — the four pages only talk through you.

Open-source DataBuff · one chat runs the same chain

• Metrics pick the target
• Topology looks downstream
• Trace splits that hop
• Logs match the original line
• → inspection report

It’s called DataBuff. Repo: https://github.com/databufflabs/databuff. Below is that chat on the official demo, watching it line the four pages up.

1 · Find the service with the highest response time

In the official demo, service-a / service-b are hitting GET /demo/checkout. The chat starts with:

找出最近 1 小时平均响应时间最高的服务，对它进行故障定位

Find the service with the highest average response time in the last hour, then localize the fault on it.

Step 1 · The AI brain hands the job to metric query

• It splits the sentence into two steps: find the slowest service in the last hour, then localize the fault on it.
• The first cut goes to metric query, time window pinned to the last hour, pull the service list first.

Step 1 · The brain sets the time window, then metric query pulls the service list.

Step 2 · Metric query names service-a

• On the table, service-a averages 240ms, the highest in this batch.
• service-b is 70ms; service-a is about 3.4× that. Fault localization stays on service-a.
• Both Java entries show 0% error rate — looks fine at a glance. MySQL error rate is 33.33%, which is high, but it isn’t the slowest, so it isn’t the main target yet.

Step 2 · Metric query names service-a; the rest of the localization stays on it.

2 · Inspect its own health, then downstream and the call chain

Step 3 · Hand fault localization to the inspection expert

• service-a itself: entry error rate 0%, no ERROR, no failed traces, 240ms is a flat line, not a sudden blast.
• The red dot is downstream [mysql]demo_apm, with 60 “entry error rate too high” alerts.
• Itself is green, the database keeps ringing, the two pages don’t match.

Step 3 · Itself is still green; downstream MySQL keeps alerting.

Step 4 · Read traces, then check the alerts

• Sampled traces are all GET /demo/checkout, status 200, duration pinned at 240ms. The entry hop shows nothing wrong.
• The database still has 60 alerts, but dependency stats write MySQL errors as 0.
• The two sentences don’t match. The only move left is to split one chain.

Step 4 · checkout is all 200; the database keeps ringing; the two sentences don’t match.

3 · Split one call, match the logs, write the report

Step 5 · Split one trace, pull the same log line

• The hop that blew up is SELECT demo_inventory, throwing InsufficientStockException.
• The entry still returns 200. The same log line matches SKU DEMO-10001, inventory left at 2.
• service-a recorded that service-b returned 200, so the entry stayed green and never went red.

Step 5 · Insufficient stock swallowed into a 200; the entry looks fine.

Step 6 · Write the inspection report

• The chat ends with an HTML inspection report, not just “done.”
• Verdict, trends, call tree, advice — the next section reads the report’s own chapters. This step doesn’t copy them first.

Step 6 · The report is out; the next section reads it by its own chapters.

4 · Open the report: verdict, trends, call tree

Start with the verdict. The report says “watch,” meaning this chain is worth looking at, not that the service is already down. 240ms is written as serial inherent latency, not a sudden spike.

You can preview this HTML report at the bottom of the chat.

Verdict “watch.” 240ms, 75%, 60 alerts, and the SKU sit on one screen.

Then the trends. All three charts are flat lines. Response time is pinned at 240ms; request rate is about 7–8/min, also flat; MySQL error rate 33.33%, also flat. This is not a one-off jitter — it was slow for the whole hour, and the alerts kept firing.

Left: 240ms flat. Right: request rate steady. Bottom: MySQL 33.33% flat.

Then the call tree. HTTP to service-b is about 100ms, the Dubbo hop about 80ms; together they are about 75% of the 240ms. The red dot lands on SELECT demo_inventory. Sum the hops and you get about 246ms, which matches the 240ms at the entry — nothing else is missing from the picture.

240ms is a serial sum; the red dot is the inventory SQL.

Then the advice. On latency, the bottleneck is two calls to service-b plus two slow SELECTs, 45ms and 30ms. On alerts, the cause is insufficient stock swallowed: SKU DEMO-10001, inventory 2. The entry returned 200 and hid it.

P0: look at the slow queries and inventory first. P1: consider parallel or merge. P2: don’t silently turn this into a 200.

5 · Same job: how long a human takes vs DataBuff’s AI analysis

The comparison is the same job: find the slowest service in the last hour, then localize the insufficient stock swallowed into a 200. Not two different incidents — one job, done twice.

The “about 3 minutes” on the right is wall-clock from this chat’s question to the report, measured. The “about 40 minutes” on the left is an estimate for a practiced human chaining the five items. There was no second stopwatch on a human run.

About 40 minutes vs about 3 minutes — roughly 13× faster.

6 · After the AI analysis, inventory, 200, and slow queries still need a human call

This chat took about 3 minutes. The slowest service is service-a, the red dot is on the database, insufficient stock was swallowed into a 200. Metrics, topology, traces, and logs lined up. Nobody had to copy a call ID across four pages.

Lined up is not closed. Whether to restock the 2 units, whether that 200 should keep swallowing this, whether those two SELECTs need a change — the report wrote advice. The call is still a human’s.

DataBuff

AI-native OpenTelemetry APM. Metrics, traces, logs, and AI troubleshooting on one stack.

GitHub: https://github.com/databufflabs/databuff

Live Demo: https://demo.databuff.ai