Why Software Factories Fail (or: harness engineering is not enough)

There's some good ideas and points in here, but this bit threw me:

> # We tried this > In July 2025 we went full lights-off

Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before.

I feel like any perspective/experience on "what agents can/can't do" from before that period is... maybe less than relevant to the modern era. TFA calls it out a few sections later with "But surely the models have gotten better since then", but then just writes off any improvement. That does not match my experience.

To me, the thing that stands out about the whole state we're in here is PR review.

Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end.

I'm not sure how the best teams do PR review, from my perspective it sucks. I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed by pulling down the branch and opening the diff with $EDITOR.

These days I think there's really no excuse for the awful UX. Linear (a company that isn't even in the domain of code review) put out a basic PR review feature[0] that is already better than what GH offers. It's simple: point a small model at the PR, group file changes together based on theme, add some commentary and sort by importance (schema changes > openapi spec).

Immediately, so much mental load has been reduced without the reviewer or the requester doing anything. This feature is pretty damn basic, and I think there are obvious next steps like generating visualisations which a dedicated product could find the time to implement.

Keen to hear others thoughts on why this is the wrong approach, or if there are tools in wide use that solve for this, or why this isnt the right problem to focus on.

0 - https://linear.app/docs/diffs#guides

>It's easy to be a little bummed by the core thesis here: "for now we're stuck reading the code".

>I was pretty excited for a world where we could just ask for things and let the models cook and not read the code and get beautiful production software

I feel so disconnected reading those things. Reading and writing code is what brings me joy. I'd never feel "bummed" or "stuck" with it.

I'm currently learning about Cost functions and Regression in the Machine learning course on coursera[1], and I cannot help but be struck by the similarities of how gradient descent seeks to minimize the error between the model and the training data, how agents do the same (far less mathematically) to seek an acceptable solution to software problems, and how light evolutionary pressures seem to guide species towards a better fit solution to the problem of existence and reproduction. (I'm no expert on the evolutionary example so go easy on me!)

[1] - this course: https://www.coursera.org/learn/machine-learning/home/welcome

I think there is a fundamental issue here of what building software even means

If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "Wait.. the API is already returning the data we need". "Wait... let's not add customers to the report who have not been active in the past year". Stuff like that

> So, why can't models do software maintainability?

I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad design?" Of course the models aren't good at it ... they're not good at anything until you've tuned them and put them in a harness that rewards good edits and throws away (improves) bad edits. That doesn't explain why "models can't do software maintainability." The real question is why harnesses can't do software maintainability, and how to build a system that can do it. (I suppose that's the purpose of the ad at the bottom of the page.)

Enjoyed most of this but unconvinced by the program design part. If I see an agent writing function signatures or listing which functions to edit in a plan that tells me that I’ve given it too big a vertical slice. I always delete the code guesses. The thing that writes the code must always spend some time discovering where/what to write or it won’t have the right context. Or put another way: “decide first, act later” always feels worse than act-learn-act.

This is one of the best writeups I've seen of this

a lot of the model's constraints come down to how they are RLed. Discussions online would be a lot better if everyone understood how the labs train the models in a high level (or did a lil data labeling)