A value with no named owner does not have a bug. It has an unfinished analysis.
👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about the six runs I spent going in a circle, and what was actually missing. Running notes live on my GitHub: github.com/brilliant-almazov.
This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.
The path in one paragraph
A user saves a set of matching rules; every domain under that account gets marked by them. Four participants: the Go API gateway as the only door, the rule-set-markup service which takes the write and folds the cascade, the domain-rule-map service which owns rule-set identity and recalculates the map, and PostgreSQL behind them - the monolith's database for the rule data, the identity service's own for identity.
One user action, four processes, two databases. And one value that decides everything downstream: which rule set is currently active for this object.
Six runs, one symptom, no code to blame
The symptom was flat and unhelpful. A rule is saved. The response is 200. The read shows the new set. The engine keeps marking by the old one.
Nothing failed. No error, no timeout, no bad payload. The verification I could do at that point was: repeat it. Eight rounds, thirty seconds apart - the read returned one identifier, the engine kept answering with another, for four minutes straight.
That defect lived through six iterations of the run - the longest of the whole period - and every one of those iterations went looking for a mistake in code. There wasn't one.
What the analysis found, once it was done
Sitting down and mapping the value took about an afternoon. Here is the whole finding.
The value lives in four carriers across two databases:
the monolith's database the engine's database
├── [1] scope row ← written by the ├── [3] engine pointer ← written by the
│ read by the door write service │ read on every engine, five
└── [2] config mirror ← written by the │ match separate paths
read by listings write service └── [3a] derived fallback — recovered from
its own map when [3] is empty
and between the two databases: no foreign key, and there cannot be one
![Two database panels: the monolith's holds [1] the scope row and [2] the config mirror, both written by the write service; the engine's holds [3] the engine pointer, written by the engine on 5 paths, and [3a] the derived fallback, not written at all; between them a dashed red line with a bolt and the words no FK](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd139myduikf4xl9x3x6q.png)
The migration that created the engine's pointer says it in a column comment, in as many words: this is a logical reference, there is no foreign key. Which is correct - the two databases belong to different services. It also means the only thing keeping four copies of one value in agreement is the order of calls and one comparison in code.
Written out as calls, the write looks like this - five steps, four of which had an owner named somewhere in a specification:
1. door → write service validate + gather touches nothing yet
2. write service → engine create identity identity row
3. write service, one tx insert scope row + mirror [1] and [2]
4. write service → engine recalculate for this object [3] ← or not
5. engine worker rebuild the map rows map rows
![A dark terminal card listing the five calls of one write, from the door to the write service through the engine worker rebuilding the map, with call 4, write service to engine, recalculate [3], highlighted and a red note: step 4, no owner](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fautp04ch8cqz1o265ooo.png)
Step 4 is the whole defect. Not its implementation - its ownership.
Second half of the finding: the right to write that value had moved seven times in 48 days.
And in one of those states - two days long - it belonged to no one.
Two correct changes, one defect
That state was produced by two commits, on the same day, in two repositories, by two different people.
- In the write service: stop naming the set on rebuild - the engine can derive which one is active.
- In the engine: the object's own active set wins over the one passed in the request.
Read either one on its own and it is an improvement. Read them together and they form a circle:
the active set ──is derived from──► the map
▲ │
│ is written only by
│ ▼
└──────────is run against────── a rebuild
A freshly written set has no map rows yet. So it can never become active - by any path at all. The rule was saved, the read showed it, and the engine had no way to reach it.
No code review inside either repository could have caught this. Each diff was right. The contradiction lived on the seam, and the seam had no owner.
Why this is an analysis failure, not a testing failure
Here is the part that stung. The class of problem was already written down - seventeen days before the first detailed run, as a marker in a verdict matrix: write ownership for this value is not settled. It was noted. It was never turned into anything that could fail.
A note that nobody has to pass is indistinguishable from no analysis at all.
What the analysis owed, and didn't deliver:
- a list of carriers of the value - all four of them, with the database each lives in;
- one named writer per carrier, and what happens when the right moves;
- a check that fails when two carriers disagree - which, being two reads compared, needs no infrastructure whatsoever;
- the same check frozen before the code that changes the ownership.
That last one is the whole point of this part. Requirement-level checks are written and frozen before the implementation moves, so that a change of ownership has something to break against.
Across the whole period this is not a small effect: of 26 numbered defects, 24 would have been caught by acceptance written before the code, and 21 of 39 defect-iterations die on a single question in the specification - who writes this value. That question is answered by reading a document, not by running anything.
What traces did, and what they didn't
Two honest halves.
Traces are what made the circle visible. Repetition proved a disagreement existed; the spans showed that the write path never called the step that moves the pointer, and that the engine resolved its own value from its own map. Without that, the argument stays "your service is wrong" / "no, yours is" - and I have lost days to that argument before.
But traces did not find the cause. They showed where the value came from at runtime. Who is allowed to put it there is not in a trace, not in a log, and not in either repository. It is a decision, and it either exists in writing or it doesn't.
What the fix actually looked like
Not a patch. A shape:
- the right to write the pointer is split into two named operations: transferring it and registering it the first time;
- the caller names the set again - whoever wrote it says which one it is;
- the pointer moves at write time, so nothing has to be rebuilt for it to take effect;
- and it was closed by both services on the same day, because the seam was fixed as a seam rather than as two separate bugs.
After that, the disagreement that had survived six iterations was cured by a single write.
The check that catches this, and why it is trivial to automate
There is no schema constraint that can span two databases. So the guard is not a constraint - it is a query, run from outside, twice:
read A the door's listing for the object → rule set = X
read B the engine's answer for the same one → rule set = Y
X != Y → the object is marked by a set it does not point at
repeat the pair 8 × 30 s: a difference that survives repetition is not a lag
Every disagreement found in the whole period came out of that shape: two answers compared, on all 392 objects rather than on a sample. It needs no new service, no CDC, no consistency framework - two HTTP reads and an equality check.
Which is exactly the kind of work worth handing to an automated executor: mechanical, wide, repeated per build, and boring enough that a person skips it.
Where automation carried it, and where it stopped
Being precise about the split, because "AI helps" on its own is not a claim worth printing.
What the executor did, and did well:
- swept all 392 objects instead of the three-object playground, per build;
- ran the two-read comparison per object and folded the disagreements into one table;
- repeated the pair eight times at thirty-second intervals to separate a real disagreement from replication lag;
- attached the trace contents - spans, durations, statuses - to every failure line, so the report survived the retention window;
- did all of it again the next morning on a new build, without being re-briefed.
What it did not do, and could not:
- decide that a value having four carriers is a problem worth naming - it had no reason to ask how many carriers exist;
- work out who is entitled to write the pointer, because that is nowhere in the running system;
- notice that two commits in two repositories, each individually sensible, formed a circle - each repository looked clean;
- turn the note in the matrix into a check that had to pass.
Every one of those is a decision, and the analysis that produces them is a person sitting down for an afternoon with the migrations and the specs. What the automation changes is the price of the consequence: once the owner is named, the check that enforces it costs two reads and runs forever.
That is the honest shape of the multiplier. Fast iteration amplifies whatever analysis exists; where analysis is missing, it amplifies the circling instead - six iterations of confidently reproducing a symptom whose cause was never in the code.
Anatomy of a service split — Part 3. Next: the other half of the same lesson - what happens when the system you compare against is being fixed at the same time, and the response code says one thing while the database holds another.
If you do this better, tell me how you record who owns a value across services. If you have been through this, what did your seam turn out to hide? If you see it differently, say where writing the check before the code costs more than it returns. How is it solved on your side, and what broke there?
