The day I asked three LLM agents to rewrite legacy Java for me — and what actually happened

1. The question that started everything

Three weeks into my internship, my supervisor sat down across from me and asked, very casually:

"OK your NLP pipeline extracts intentions and rules from legacy Java. Nice. And then what?"

I looked at him. I looked at my laptop. I looked back at him.

The whole project — Pulsar Modernizer — was supposed to eventually turn legacy Java into modern Spring Boot code. My part was the "understand the old code" part. F1 = 0.857 on the annotated corpus, a shiny React UI, everything humming in Docker.

But the "and then?" was doing a lot of work in that sentence.

That evening I wrote in my notes: "Nobody has actually tried the generation part. Everyone assumes it'll be easy because LLMs. That is very obviously wrong."

So I decided to try.

2. Why "just prompt an LLM to rewrite it" doesn't work

The naive move — feed the old code and the extracted rules to an LLM and say "please modernize this" — has three problems and I hit all of them in the first hour:

1. The model hallucinates. It happily invents helper classes that don't exist and calls methods with the wrong signature.
2. You have no criterion for stopping. The model tells you "it's done ". OK. Is it? By what test?
3. You have no criterion for equivalence. Even if it compiles, how do you know the new code actually does what the old one did?

I needed something more constrained than "prompt it and pray".

3. The setup — a chain, not a monolith

I ended up building three specialized agents in sequence:

IntentCard + RuleCards
        │
        ▼
[APIDesigner] ──► JSON contract (class, methods, DTOs, throws)
        │
        ├───────────────┐
        ▼               ▼
[CodeGenerator]    [TestGenerator]
        │               │
        ▼               ▼
     .java          *Test.java
        │               │
        └────► verifier (mvn test)

The key insight: each rule extracted from the legacy code should become a test that the generated code has to pass.

This flips the whole thing. I don't trust the LLM. I trust javac and JUnit.

I did all of this on a local model — Qwen 2.5 Coder 3B via Ollama. No cloud APIs, no data leaving my Mac. On a 3B model, the constraint is fair: if the pipeline needs GPT-4-level intelligence to hold together, it's not going to fly in a bank in production.

4. The three cracks I never expected

Here's where it got interesting.

I hand-wrote the JUnit test-oracles for four rules from my banking corpus. Then I asked the LLM to generate the code. Baseline result:

• 2 out of 3 oracles green with a "raw" prompt.
• 3 out of 3 with a "clarified" prompt.

The difference between the two prompts was three conventions I added at the top:

1. null inputs are always rejected (never silently ignored)
2. "cannot exceed N" means N is accepted, N + ε is rejected
3. Currency/country lists are case-sensitive by default

Every failure I observed traced back to one of these three assumptions being different in my head vs the LLM's head.

Which means: the three cracks were in my own annotations, not in the model. My RuleCards were technically valid English sentences that a human reader would interpret one way, and a language model would interpret the other way. Nobody had written down the tiebreaker.

That was the aha moment of the whole internship. Not "LLMs are amazing". Not "LLMs are terrible". Just: your dataset has hidden ambiguities, and a fluent model surfaces them by making the opposite choice from what you meant.

5. The Markov-1 pathological fixed point

I got greedy. I built a repair loop: if the generated code doesn't compile, feed the compiler error back into the prompt and ask the model to fix it. Up to 5 iterations.

Round 1: on a simple case (DevisePermise — accepted currencies), the 3B model produced code that treated null as .toUpperCase() and crashed at runtime. I fed the error back. Round 2: fixed. 2 iterations, converged. Beautiful.

Then I threw a harder rule at it: LimiteVelociteTransactions — count more than 20 transactions in a sliding hour window, add +30 to the fraud score.

Round 1: if (nowTimestamp == null) — but nowTimestamp is a Java long primitive. long can't be null. javac refused. I fed the error back.

Round 2: same code. Same error.
Round 3: same. Same.
Rounds 4, 5: exactly the same generated code.

The loop had converged — on broken code. The 3B model, presented with a clear javac message, could not figure out that long isn't Long. And because my repair prompt was Markov-1 (only the previous attempt), it kept producing the same fixed point.

The takeaway: the fact that a repair loop "converges" tells you nothing about whether it converges on something correct.

6. What I actually shipped

Full disclosure — this isn't a production system. It's a slice, on 4 rules, in a language (Java) that has parsers everywhere, with a corpus I annotated myself. The results are directional, not statistical.

But the plumbing works end-to-end. A user pastes a Git URL in the interface. The system clones, extracts intentions with the NLP pipeline, persists them in PostgreSQL as a Cognitive Knowledge Graph, and — from a single click — chains the three agents to produce a downloadable zip of modern Spring Boot code. On Spring PetClinic (30 files), the loop currently produces four working Java classes with tests in about 10 minutes.

You can compile the zip. You can read the tests. You can also see, in the output, that the model sometimes invents helper types the contract never mentioned. That's fine. That's what the human-in-the-loop is for.

7. What I'd do differently

Three things that I already know I got wrong and would fix on day one of a v2:

1. Escalate to a bigger model after N repair failures. 3B is fast; 7B has 4× the reasoning depth. Cascade them.
2. Use a validator of attribution. Before generating tests, check that every method signature the LLM invents is in the APIDesigner's contract. Reject and re-ask if not.
3. Amend the annotation doctrine with three explicit fields (null_policy, bound_inclusivity, case_sensitive) instead of relying on three "conventions" in the prompt. Right conventions in the wrong place.

8. The one thing I'm actually taking away from this

Every tutorial I read before starting this project was about how to prompt an LLM to write good code. None of them talked about how to know if the code is good.

That's the whole game. The LLM is the cheap part. The evaluator, the oracle, the human-in-the-loop, the annotation doctrine — that's where all the actual work lives.

A 3B model with a well-designed loop beats a 70B model with none.

Meryeme Ramdi is a second-year AI engineering student at ENSIAS, currently interning on the Pulsar Modernizer project at Pulsaride Solutions. She writes about ML systems that ship, at dev.to/meryyy.