What happens when you submit a solo open-source project to academic peer review:
the reviews were good, the paper was rejected, and the gap they found was the one
I had no way to close on my own.
I've been building DoSync,
an open-source protocol that sits between AI agents and physical devices — I
wrote about
why it exists here
if you want the background.
I tested it the only way I could: on my own hardware, against my own
expectations. Then I sent it to IEEE WF-IoT 2026, because I wanted it examined by
people who weren't me.
Five reviews came back. Four were substantive. It was rejected.
And here's the part that took me longest to understand: the reviews were
good. Two scored relevance 5 out of 5. One recommended minor revisions. The
words included "well-written", "highly pragmatic", "a clear and defensible
architectural stance", and — the one I keep re-reading — "unusually candid about
limitations."
Rejected anyway. Not because the idea was wrong. Because of this, from the most
detailed review:
The evaluation is small and self-judged: 15 scenarios with ground truth
defined by the author alone.
That sentence is correct, and no amount of rewriting section 4 was going to
change it.
I got exactly what I asked for
It took me a few days to notice that the rejection was the thing I'd submitted
for.
I didn't send the paper hoping for a stamp. I sent it because after months of
being the only person who had ever evaluated this, I had no way to tell the
difference between "this works" and "this works the way I expect it to." Four
strangers read it carefully and told me. That is not a consolation prize — it is
the entire reason to submit.
What they said, in summary: the architecture holds up, and the evidence doesn't
reach as far as the claims. One reviewer went further and put the knife somewhere
specific. My central claim is that the project removes the manual burden of
writing device-by-device automation rules. Their objection:
if resolution quality depends on well-curated tags, the manual configuration
burden has not been eliminated, it has moved from rules to manifest curation
I've had months to find a counter-argument. I don't have one yet.
The one thing I couldn't fix by myself
The request that came up in almost every review was for independent annotators.
Not more scenarios — different people. Someone other than me who had read a
scenario, looked at the devices available, and formed their own opinion about
which ones should respond.
I couldn't produce one. Every judgement about what the resolver should pick —
in all fifteen scenarios, in the paper and in the benchmark files — was mine.
When I disagreed with the resolver, I was the one who had decided what the right
answer was in the first place.
Everything else on the reviewers' list, I could work on alone, and did. The
evaluation used to run on a single home registry, which is a poor way to support
a claim of being domain-agnostic; there's now a
multi-domain corpus
covering industrial and clinical registries, published so the numbers are
reproducible by someone who isn't me. The precision/recall conventions that made
two rows of a results table mathematically impossible are defined and the bug is
traced. There's a sensitivity analysis on the scoring weights.
None of that required anyone else. The annotators do. That asymmetry turned out
to be the most useful thing in the whole review packet.
Distribution is harder than building, and worth more
Since the reviews, I've been doing the closest thing to third-party testing that
one person can do: installing my own project on machines that had never run it,
following my own published instructions, on hardware that owed me nothing.
It found more than a dozen defects in a week, on a codebase with about a thousand
passing tests. My favourite: device discovery had been silently dead in
production for days, because a dependency I'd added shipped an event loop that
declares a method and raises when you call it. My check was hasattr. It
returned True. Every test passed and nothing was ever discovered.
Not one of those was findable from the inside. My test suite ran on one operating
system, one event loop, one machine that already had everything configured — it
was measuring a world only I lived in. Which is, word for word, what the reviewer
said about my evaluation.
That's the part I'd pass on to anyone building something alone: the hardest
work isn't the next feature. It's getting the thing into the hands of someone who
doesn't already know how it's supposed to behave — including your own hands, on
a machine that has never seen it.
The protocol is open, the corpus is public, and the one thing it still needs is
an opinion that isn't mine.
GitHub: github.com/giulianireg-spec/dosync-protocol
Web: https://dosync.dev/
License: Apache 2.0
DoSync — the semantic layer between AI agents and physical systems.
(The reviews are summarised faithfully and the reviewers stay anonymous, as they
should.)
