On-Policy vs Off-Policy Training of LLMs: How Models Start Learning From Their Own Outputs

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

There is a simple question at the heart of modern LLM training:

Where did the data come from?

If you train a model on answers generated by itself, you are doing something fundamentally different from training it on answers generated by an older model, a human, or a static dataset.

That difference is called on-policy vs. off-policy learning.

It sounds like reinforcement-learning terminology from the 1990s. It is. But it has become one of the most important ideas for understanding what is happening in modern LLM post-training: RLHF, PPO, rejection sampling, preference optimization, self-play, synthetic data, verifiable rewards, and increasingly, models that generate their own training trajectories.

The interesting part is that the distinction is not really about "online" versus "offline" data.

It is about a much more precise question:

Is the model learning from behavior produced by the policy it is currently trying to improve?

Once you see that distinction, a surprising number of LLM training techniques fall into place.

1. Imagine teaching a programmer

Suppose you are training an LLM to write Python.

You give it:

Write a function that returns the longest increasing subsequence.

The model produces:

def lis(a):
    ...

An evaluator gives it a score.

Now imagine two training systems.

System A: on-policy

The current model generates the solution.

You evaluate that solution.

You update the model based on the result.

Then the new model generates another solution.

The loop is:

current model
     |
     v
generate solution
     |
     v
evaluate solution
     |
     v
update model
     |
     v
new model
     |
     +----> generate again

The data continuously moves with the model.

System B: off-policy

Instead, you have 10 million solutions sitting in a dataset.

Some were written by humans.

Some came from GPT-4.

Some came from an older checkpoint.

Some came from a specialized coding model.

You train your current model on those examples.

The behavior that produced the data is not necessarily the behavior of the model you are currently training.

That is off-policy learning.

The distinction matters because the model's mistakes determine what it gets to learn from.

If the current model is terrible at recursion, an on-policy system will naturally generate lots of terrible recursive solutions. An off-policy dataset might contain excellent recursive solutions that the current model would never have generated.

That sounds like an obvious advantage for off-policy learning.

And sometimes it is.

But there is a catch.

2. The RL concept is older than LLMs

The terminology comes from reinforcement learning.

Chris Watkins' Q-learning work in the late 1980s and early 1990s gave one of the classic examples of off-policy learning. Q-learning can learn about an optimal policy while the agent is actually behaving according to another policy.

The conceptual trick is powerful:

The policy generating the experience does not have to be the policy being learned.

Contrast that with policy-gradient methods, where you typically generate trajectories using the current policy and then use those trajectories to estimate how changing that policy would affect expected reward.

This distinction became especially important as reinforcement learning moved from toy environments to expensive neural-network systems.

By the time John Schulman and colleagues introduced PPO in 2017, the engineering problem was familiar:

Generate experience with a policy, then update the policy without moving it so far that the experience becomes useless.

PPO explicitly alternates between collecting samples from the current policy and optimizing on those samples. It permits multiple optimization epochs over the collected data while constraining how far the updated policy moves from the policy that generated the samples.

That constraint is not cosmetic.

It is the central operational problem.

Suppose your model generated:

"The answer is 42."

You update the model ten times using that batch.

After those ten updates, your model may have changed substantially.

The sample was generated by policy P_old.

You are now optimizing policy P_new.

The more different P_new becomes from P_old, the less directly the old sample tells you about P_new.

That is the basic tension behind on-policy RL.

3. Why LLMs make this unusually expensive

In a game like Atari, generating another million actions may be relatively cheap.

For an LLM, generating another million trajectories can mean running a giant transformer for billions of tokens.

And the reward might require an expensive evaluator.

Consider a deliberately simple calculation.

Suppose:

• model inference costs $2 per million generated tokens
• you generate 100 million tokens per training iteration
• you perform 100 iterations

Generation alone costs roughly:

100M tokens x 100
= 10B generated tokens

10B / 1M x $2
= $20,000

That is a toy number, but the scaling relationship is real.

At frontier-model scale, the expensive resource is often not the gradient update.

It is producing useful experience.

This is why off-policy learning is so attractive.

If you have already paid to generate 10 billion tokens, you would very much like to reuse them.

And you would like to reuse them more than once.

That is the economic argument for off-policy training:

Experience is an asset. Don't throw it away after one gradient update.

This is also why replay buffers became such an important idea in classical RL.

But LLMs introduce an even more interesting problem: the "environment" is often other models, humans, tools, or verifiers rather than a game simulator.

4. InstructGPT shows the transition

One of the clearest historical examples is OpenAI's 2022 InstructGPT work.

The basic pipeline was:

GPT-3
  |
  v
human demonstrations
  |
  v
SFT model
  |
  v
generate multiple answers
  |
  v
human preference rankings
  |
  v
reward model
  |
  v
PPO
  |
  v
InstructGPT

The important part for our discussion is the final stage.

The reward model evaluates outputs generated by the policy, and PPO uses those interactions to improve the policy.

That is much closer to the classic on-policy RL loop than ordinary supervised fine-tuning.

And it produced a remarkable result.

OpenAI reported that evaluators preferred outputs from the 1.3B-parameter InstructGPT model over the 175B-parameter GPT-3 model on their instruction-following evaluation.

In other words, changing how the model learned from experience could matter more than making the model roughly 100x larger.

This was an important moment in LLM history because it demonstrated that post-training was not merely polishing a pretrained model.

It could substantially change what the model optimized for.

And PPO's on-policy nature was part of that story.

5. So why not always use on-policy learning?

Because on-policy learning has a brutal property:

The data expires quickly.

Imagine training version 1 of your model.

It generates:

Prompt: Prove that sqrt(2) is irrational.

Answer: ...
Reward: 0.8

You update the model.

Now you have version 2.

Why should version 2 be restricted to learning from version 1's trajectories?

It might be able to solve the problem much better.

Conversely, version 1 may have generated a brilliant proof that version 2 will almost never discover again.

This creates a strange asymmetry.

On-policy learning gives you highly relevant data:

data ~= current behavior

but potentially wastes enormous amounts of useful historical data.

Off-policy learning gives you reusable historical data:

data != necessarily current behavior

but now you have to deal with the distribution mismatch.

At a high level:

On-policy:
    fresh + relevant
    expensive + disposable

Off-policy:
    reusable + diverse
    potentially stale + mismatched

This is one reason modern LLM training increasingly looks like a hybrid system rather than a pure on-policy or pure off-policy system.

You want the freshness of on-policy data and the economics of off-policy data.

6. The math: what exactly goes wrong?

Let the model be a policy:

pi_theta(y | x)

This means:

Given prompt x, what probability does the model with parameters theta assign to answer y?

Suppose the model gets reward R(x, y).

The objective is conceptually:

J(theta) = E[R(x, y)]

where y is sampled from the model itself.

For a policy-gradient method, a basic gradient estimator looks like:

grad J(theta)
    ~= E[ grad log pi_theta(y | x) * R ]

The important thing is that y was sampled from:

pi_theta

Now suppose we have old data generated by another policy:

pi_old

but we want to optimize:

pi_theta

The expectation is now being taken under the wrong distribution.

One classical solution is importance sampling.

Conceptually:

E_pi_theta[f(y)]

    =
E_pi_old[
    pi_theta(y|x) / pi_old(y|x) * f(y)
]

The ratio

pi_theta(y|x) / pi_old(y|x)

corrects for the fact that the data came from the old policy.

This looks elegant.

It can also become horrible.

Suppose a sequence has probability:

pi_old(y|x) = 1e-6

under the old model but

pi_theta(y|x) = 1e-3

under the new model.

The importance weight is:

1e-3 / 1e-6 = 1000

One example can suddenly have 1,000 times the influence of another.

With long autoregressive sequences, probability ratios can become extremely volatile because token-level ratios multiply across the sequence.

That creates a classic RL engineering problem:

How much old data can we safely reuse before the correction becomes statistically ugly?

PPO takes a pragmatic route.

Instead of allowing arbitrary policy movement, it clips the probability ratio:

r(theta) =
    pi_theta(y|x) / pi_old(y|x)

and uses an objective that effectively says:

Improve the policy, but don't benefit too much from moving far away from the policy that generated this experience.

This is one reason PPO became so popular: it turns a theoretically nasty distribution-shift problem into something engineers can actually operate.

7. The LLM world is now blurring the boundary

Here is where things get interesting.

People often casually say:

"DPO is off-policy RL."

That is useful shorthand, but technically it is better to be precise.

DPO, introduced by Rafael Rafailov and colleagues in 2023, takes preference pairs such as:

prompt
chosen answer
rejected answer

and directly optimizes the language model using those preferences.

There is no PPO rollout loop.

No reward-model inference is required during optimization.

No requirement that the current model generate every training example.

So operationally it looks much more like learning from a fixed preference dataset.

This is one reason DPO was attractive: it removed much of the machinery associated with RLHF while retaining a principled connection to the underlying reward-maximization problem.

But now consider what happens if we repeatedly generate preference data using the current model:

Model v1
   |
   v
generate candidates
   |
   v
judge / verifier
   |
   v
preference dataset
   |
   v
train Model v2
   |
   v
generate candidates
   |
   v
judge / verifier
   |
   v
train Model v3

You have built something that is neither simply "offline training" nor simply classical on-policy PPO.

You have a data-generation loop whose policy evolves over time.

That distinction is becoming increasingly important for reasoning models.

A model might generate thousands of candidate proofs, programs, mathematical solutions, tool-use trajectories, or chains of actions. A verifier selects successful ones. Those successful trajectories become training data.

The resulting system has an economic structure very different from ordinary SFT:

compute
  -> generate attempts
  -> evaluate attempts
  -> retain valuable experience
  -> train
  -> generate better attempts
  -> repeat

The bottleneck can move from gradient computation to experience generation and evaluation.

That is the deeper reason on-policy versus off-policy matters for LLM developers.

It is not merely a taxonomy of RL algorithms.

It is a question about how efficiently you turn inference compute into learning signal.

8. The practical rule for LLM engineers

A useful mental model is:

The most important engineering questions therefore become:

1. Who generated this data?

Not just "is it synthetic?"

A dataset generated by your current checkpoint is very different from one generated by a model six generations ago.

2. How far away is the behavior policy?

If your current model assigns very different probabilities to the training trajectories, you have distribution shift.

3. How expensive is experience?

If generation is cheap, throwing data away may be fine.

If generation requires a giant reasoning model plus a verifier, replay becomes much more attractive.

4. How reliable is the reward?

On-policy learning can repeatedly exploit a flawed reward function.

The model gets better at finding whatever the evaluator rewards, rather than what you actually wanted.

5. Is diversity valuable?

Off-policy datasets can contain behaviors that the current model would never discover.

This can be extraordinarily valuable in reasoning and coding.

Imagine your model has a 0.01% probability of discovering a particular algorithm.

An on-policy system might need an enormous number of rollouts to find it.

An external expert, stronger model, or historical checkpoint might already have produced it.

In that situation, insisting on on-policy data is throwing away information.

The interesting design space is therefore not:

"Should I use on-policy or off-policy training?"

It is:

Which experiences should be generated by the current policy, which should be harvested from other policies, and how aggressively should each kind be reused?

That is a much more useful question.

And it points toward a future where the training system itself looks increasingly like an experience-management system:

                  +------------------+
                  |  Current model   |
                  +--------+---------+
                           |
                     generate
                           |
                           v
                    +-------------+
                    |  Evaluator  |
                    +------+------+ 
                           |
                 +---------+---------+
                 |                   |
                 v                   v
          fresh experience      replay buffer
                 |                   |
                 +---------+---------+
                           |
                           v
                    policy update
                           |
                           v
                     new model

The winning systems may not be the ones that are purely on-policy or purely off-policy.

They may be the ones that are best at deciding when fresh experience is worth paying for and when old experience is still valuable.

That is ultimately an economics problem disguised as a reinforcement-learning problem.

And for LLMs, the economics are unusually stark: every trajectory is potentially an expensive experiment, and every successful trajectory is potentially reusable training capital.

9. The takeaway

The simplest way to remember the distinction is:

On-policy learning learns from what the model currently does. Off-policy learning learns from what some behavior did.

On-policy training gives you data that is tightly matched to the policy being optimized, but generating that data can be enormously expensive.

Off-policy training lets you reuse experience, mix data from different models, and learn from behavior the current model might never discover. But now you inherit the statistical problem of distribution mismatch.

Classical RL encountered this decades ago with Q-learning, actor-critic methods, and policy-gradient algorithms. LLMs have simply made the underlying tradeoff vastly more expensive and more consequential.

InstructGPT demonstrated the power of policy optimization for language models. PPO provided a practical mechanism for repeatedly improving a policy from its own rollouts. DPO subsequently showed how much of the preference-learning problem could be reformulated as direct optimization on a fixed dataset.

The next step is arguably more interesting.

Once models can generate enormous amounts of candidate reasoning, code, proofs, tool trajectories, and other experiences—and automated evaluators can decide which experiences are valuable—the central training question becomes:

How should an LLM decide which of its experiences to learn from, how many times to reuse them, and when to spend compute generating new ones?

That sounds less like traditional fine-tuning and more like building a learning system with a memory.

Do you think future frontier-model training will converge toward predominantly on-policy learning, or will replaying and intelligently curating experience become the bigger competitive advantage?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.