Self-Play Data Factory

Turn one LLM into a training-data factory. A single model generates N diverse candidate answers, scores each one with a fine-grained A–T letter-scale logprob verifier, and emits a clean DPO preference dataset — no second model, no human labels, no binary "good vs bad".

Self-Play Data Factory

Why this exists

Training-data generation for DPO/RLHF usually needs a second, stronger
model as a judge, or a human in the loop. This project shows you can get
surprisingly good preference data from one model playing both roles:

Actor — generates N candidate answers to a prompt.
Critic — the same model scores each candidate on a fine-grained 20-letter scale, and we read the token-logprob distribution (not just the sampled letter) to recover a calibrated 0–1 reward.

The trick that makes it work is the letter-scale logprob expectation
(replicated from the LLM-as-a-Verifier
technique). Ask a model to "rate 1–10" and you get a coarse, noisy integer.
Ask it for one letter token and read the softmax over the letters it was
torn between — you get a fine-grained reward (0.842 vs 0.860) that lets
the factory pick genuinely better candidates and emit clean preference
pairs, instead of arbitrary good-vs-bad flips.

Features

N candidate generation with a temperature sweep (0.3–1.2) for diversity and exact-duplicate dedup.
A–T letter-scale verifier (20 letters) that reads top_logprobs: 20 and computes the expectation over the letter distribution.
Criteria decomposition — scores each candidate against multiple criteria (Correctness, Clarity, Efficiency, …), user-defined or auto-generated from the prompt.
Repeated evaluation (--n-evaluations N) to average out variance.
DPO preference pairs — chosen = top candidate, rejected = bottom, emitted only when the score gap is meaningful (Δ > 0.05 default).
Zero SDK, zero framework — global fetch only. Talks to any OpenAI-compatible endpoint: DeepSeek, OpenAI, vLLM, Ollama, LM Studio.
Deterministic mock LLM — runs end to end with no API key, while still exercising the full logprob path.
Friendly CLI — a helpful hint (not a crash) when no key is present.
Unit-tested math — the expectation core is pinned down with node:test.

Architecture

                         ┌─────────────────────────────┐
                         │       src/index.ts (CLI)    │
                         │  --mock --prompt --candidates│
                         │  --criteria --n-evaluations │
                         └──────────────┬──────────────┘
                                        │
                                        ▼
                         ┌─────────────────────────────┐
                         │      src/factory.ts         │
                         │  runFactory(): orchestrates │
                         └───────┬──────────┬──────────┘
                                 │          │
                 generate        │          │  verify
                                 ▼          ▼
         ┌───────────────────────┐   ┌─────────────────────────────┐
         │   src/generator.ts    │   │       src/verifier.ts       │
         │ N calls, temp sweep,  │   │ A–T letter-scale logprob    │
         │ exact dedupe          │   │ expectation scorer          │
         └───────────┬───────────┘   └──────────────┬──────────────┘
                     │                              │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                     ┌─────────────────────────────┐
                     │   src/llm.ts                │
                     │ OpenAICompatibleClient      │
                     │ chat() + chatWithLogprobs() │
                     │ (global fetch, no SDK)      │
                     └──────────────┬──────────────┘
                                    │
                    POST /v1/chat/completions
                                    │
                          ┌─────────▼─────────┐
                          │  Any OpenAI-      │
                          │  compatible       │
                          │  endpoint         │
                          │ (DeepSeek/vLLM/   │
                          │  Ollama/LM Studio)│
                          └───────────────────┘

The data flow

prompt
  │
  ▼ 1. generateCandidates(prompt, N, {temperature sweep})
  ▼
N candidates ──► dedupe exact duplicates
  │
  ▼ 2. resolveCriteria (explicit --criteria or auto-generated)
  │
  ▼ 3. scoreCandidates(prompt, candidates, criteria)
  │       per (candidate, criterion):
  │         chatWithLogprobs → first-token top-logprobs (A–T)
  │         expected = Σ P(letter) · score(letter)   ← fine-grained reward
  │         averaged over N evaluations
  │
  ▼ 4. rank descending by score
  │
  ├──► 5. print leaderboard (score + per-criterion + snippet)
  │
  └──► 6. buildPairs: chosen = top, rejected = bottom
            emit only when score_delta > minScoreDelta (default 0.05)
          │
          ▼ 7. write data/dataset.jsonl (DPO format)

Technologies

Layer	Choice	Why
Language	TypeScript (ESM, strict)	Type safety across the pipeline; `.ts` extension imports, `verbatimModuleSyntax`, `noUnusedLocals`
Runtime	Node 18+	Global `fetch` and `AbortController` — no HTTP SDK
Runner	tsx	Run TypeScript directly, zero build step
HTTP	global `fetch`	Talks to any OpenAI-compatible `/v1/chat/completions`
Config	hand-rolled `.env` parser	No `dotenv` dependency (splits lines on `=`)
Tests	`node:test` (built-in)	No test framework dependency
Dev deps	`typescript`, `tsx`, `@types/node`	The only three

Dependencies (runtime): zero. package.json has an empty
dependencies block — the entire project runs on Node's built-ins.

Project layout

selfplay-factory/
├── package.json          # type:module; scripts: demo, demo:mock, typecheck, test
├── tsconfig.json         # strict, ESNext, moduleResolution bundler, noEmit
├── .env.example          # OPENAI_BASE_URL / OPENAI_API_KEY / OPENAI_MODEL / VERIFIER_MODEL
├── .gitignore            # node_modules, .env, data/*.jsonl, etc.
├── README.md             # this file
└── src/
    ├── config.ts         # manual .env parser → Config
    ├── llm.ts            # OpenAICompatibleClient: chat() + chatWithLogprobs()
    ├── generator.ts      # generateCandidates: N calls, temp sweep, dedupe
    ├── verifier.ts       # THE core: A–T letter-scale logprob expectation
    ├── prompts.ts        # generator + verifier-rubric + criteria-gen templates
    ├── factory.ts        # orchestration → rank → pairs → dataset.jsonl → leaderboard
    ├── mock.ts           # deterministic mock LLM with a full logprob distribution
    ├── index.ts          # CLI entry
    └── verifier.test.ts  # node:test unit tests for the expectation math

Quickstart

npm i

# Demo with the deterministic mock LLM — no API key needed.
npm run demo:mock
# or
npx tsx src/index.ts --mock --prompt "Write a hello world" --candidates 5 --n-evaluations 1

That prints a leaderboard and writes data/dataset.jsonl:

{"prompt":"Write a hello world","chosen":"Here is a robust hello world: ...","rejected":"I don't know. I cannot answer this question.","score_chosen":0.842,"score_rejected":0.004,"score_delta":0.838}

Run the tests and typecheck:

npm test          # node:test — 16 unit tests
npm run typecheck # tsc --noEmit — must pass clean

Point it at a real model (DeepSeek v4 flash)

cp .env.example .env

OPENAI_BASE_URL=https://api.deepseek.com/v1
OPENAI_API_KEY=sk-…
OPENAI_MODEL=deepseek-v4-flash-0731
VERIFIER_MODEL=deepseek-v4-flash-0731   # optional; defaults to OPENAI_MODEL

Then:

npm run demo     # real endpoint, 5 candidates, 1 evaluation

Any OpenAI-compatible endpoint works — DeepSeek, OpenAI, vLLM
(http://localhost:8000/v1), Ollama (http://localhost:11434/v1), LM Studio.

No key + no --mock? The CLI prints a friendly hint instead of
crashing. It also explains exactly how to point at DeepSeek v4 flash.

CLI reference

npx tsx src/index.ts [flags]

--mock                 Use the deterministic mock LLM (no API key needed)
--prompt "…"           Prompt to generate candidates for
--candidates N         How many candidates to generate        (default 5)
--criteria "A,B,C"     Explicit criteria; omit to auto-generate
--n-evaluations N      Average N repeated scores per criterion (default 1)
--min-score-delta X    Min score gap to emit a pair             (default 0.05)
--out-file PATH        Output path                              (default data/dataset.jsonl)
-h, --help             Help

Example:

npx tsx src/index.ts --mock --prompt "Write a fibonacci function" \
  --candidates 4 --criteria "Correctness,Clarity,Efficiency" --n-evaluations 3

The verifier algorithm — the heart of it

Naive scoring asks the model to "rate 1–10" and trusts the single sampled
integer — a coarse, noisy, hard-to-calibrate signal. This project replicates
the LLM-as-a-verifier technique:

1. Letter scale A–T (20 letters), not digits

The verifier is told to answer with exactly one letter. Letters force a
clean single-token distribution. The prompt (src/prompts.ts):

# Reply format
Reply with exactly one letter. No punctuation, no explanation.

Available letters:
  A
  B
  ...
  T

2. Logprob extraction

Every verification call sends logprobs: true, top_logprobs: 20 and reads
the first token position's top-logprob list (src/llm.ts):

async chatWithLogprobs(message: ChatMessage, opts: ChatOptions = {}) {
  const data = await this.post(this.buildBody(message, { ...opts, logprobs: true }));
  const firstToken = data?.choices[0]?.logprobs?.content?.[0] ?? null;
  const topLogprobs = firstToken?.top_logprobs ?? null;
  return { text: content.trim(), topLogprobs };
}

The CLI logs what arrives, so you can watch the raw distribution:

[llm] first token="B" top=["B","A","C"]

3. Expectation over the distribution

Each observed letter maps to a score in [0,1], evenly spaced:
A = 1.0, B = 1 − 1/19, …, T = 0.0 (scoreLetter):

export function scoreLetter(letter: string | null): number {
  const idx = LETTERS.indexOf(letter.toUpperCase());
  return SCORE_MAX - (idx / (LETTER_COUNT - 1)) * (SCORE_MAX - SCORE_MIN);
}

The score is the expected value over the letter distribution
(expectedScoreFromLogprobs):

expected = Σ P(letter) · score(letter)

where P(letter) is the softmax of the letter's logprob, renormalized over
the letters that actually appeared (out-of-scale tokens like U or
multi-char tokens are dropped):

export function expectedScoreFromLogprobs(topLogprobs: TokenLogprob[] | null): number | null {
  // ...collect the A–T letters that appeared with their logprobs...
  let z = 0;
  for (const s of seen) {
    s.p = Math.exp(Math.log(s.p) - logSum); // normalize by max-logprob
    z += s.p;
  }
  let expected = 0;
  for (const s of seen) {
    expected += (s.p / z) * scoreLetter(s.letter);
  }
  return expected;
}

If the model is torn between B and A, the score lands between 0.947
and 1.0 instead of flipping a coin — that's the fine-grained signal that
makes the factory work.

4. Criteria decomposition

Each candidate is scored against multiple criteria (Correctness,
Clarity, Efficiency, …), user-provided or auto-generated from the prompt.
The final score is the mean over criteria:

const scores = details.map((d) => d.score).filter((s) => !Number.isNaN(s));
const final = scores.length > 0 ? scores.reduce((a, b) => a + b, 0) / scores.length : NaN;

5. Repeated evaluation (optional)

--n-evaluations N runs each (candidate, criterion) pair N times and
averages, reducing per-sample variance:

const score = used.length > 0 ? used.reduce((a, b) => a + b, 0) / used.length : Number.NaN;

Fallbacks

Some endpoints ignore logprobs. If no logprob list comes back, the verifier
parses the letter from the text; if there's no letter, it parses a plain
0–10 / 0–1 number. All of this is unit-tested.

The factory pipeline

src/factory.ts orchestrates the whole thing:

export async function runFactory(prompt: string, opts: FactoryOptions): Promise<FactoryResult> {
  // 1. Generate candidates.
  const candidates = await generateCandidates(prompt, nCandidates, { client, model, log });

  // 2. Resolve criteria (explicit or auto-generated).
  const criteria = await resolveCriteria(client, cfg, prompt, opts.criteria ?? [], log);

  // 3. Verify every candidate.
  const scored = await scoreCandidates(prompt, candidates, criteria, { client, model, nEvaluations, log });

  // 4. Rank descending.
  const ranked = [...scored].sort((a, b) => b.score - a.score);

  // 5. Build preference pairs (chosen = top, rejected = bottom).
  const pairs = buildPairs(prompt, ranked, minDelta);

  // 6. Write data/dataset.jsonl (DPO format).
  writeFileSync(resolve(outFile), lines.join('\n') + '\n', 'utf8');

  // 7. Print the leaderboard.
  printLeaderboard(ranked, criteria);
}

Pair-building is deliberately conservative — a pair is emitted only when
the score gap is meaningful:

export function buildPairs(prompt: string, ranked: ScoredCandidate[], minDelta: number) {
  const top = ranked[0];
  const bottom = ranked[ranked.length - 1];
  const delta = top.score - bottom.score;
  if (delta <= minDelta) return [];   // no meaningful difference → no pair
  return [{ prompt, chosen: top.candidate, rejected: bottom.candidate,
            score_chosen: top.score, score_rejected: bottom.score, score_delta: delta }];
}

DPO dataset format

Each row of data/dataset.jsonl is directly usable by DPO/RLHF training
loops:

{
  "prompt": "Write a hello world",
  "chosen": "Here is a robust hello world: ...",
  "rejected": "I don't know. I cannot answer this question.",
  "score_chosen": 0.842,
  "score_rejected": 0.004,
  "score_delta": 0.838
}

The mock LLM

src/mock.ts is a deterministic stand-in that still exercises the full
logprob path: for every verification call it emits a 20-letter logprob
distribution whose mass concentrates around the "true" quality of the
candidate — so the expectation math runs exactly as it does against a real
endpoint. It also models temperature diversity (low temperature → well-formed
answer, high temperature → evasive), so --mock demos the whole factory
offline.

Unit tests

src/verifier.test.ts pins down the core math with node:test (no
framework). Highlights:

test('expectedScore: degenerate distribution on A → 1.0', () => {
  const row = logprobRow([['A', 0], ['B', -100]]);
  assert.equal(expectedScoreFromLogprobs(row), 1.0);
});

test('expectedScore: uniform over A..T → 0.5', () => {
  const row = logprobRow(LETTERS.map((l) => [l, 0]));
  assert.ok(Math.abs(expectedScoreFromLogprobs(row)! - 0.5) < 1e-12);
});

test('expectedScore: renormalization ignores non-letter tokens', () => {
  // "U" has the highest logprob but is out-of-scale → must be dropped.
  const row = logprobRow([['U', 0], ['A', -1], ['T', -1]]);
  assert.ok(Math.abs(expectedScoreFromLogprobs(row)! - 0.5) < 1e-9);
});

Run with npm test (16 tests, all passing).

Forking & contributing

Get set up

git clone <your-fork-url>
cd selfplay-factory
npm i
npm test          # 16 tests should pass
npm run typecheck # tsc --noEmit should be clean

If npm i hits a permission error on the global npm cache, use a
project-local cache: npm install --cache "$(pwd)/.npm-cache".

Where to look first

If you want to change…	Start here
The scoring math	`src/verifier.ts` + `src/verifier.test.ts`
The verifier prompt / rubric	`src/prompts.ts`
Candidate generation	`src/generator.ts`
Orchestration / output format	`src/factory.ts`
The HTTP client	`src/llm.ts`
Mock behavior	`src/mock.ts`
CLI flags	`src/index.ts`

Contribution workflow

Fork the repo and create a feature branch.
Add or update unit tests in src/verifier.test.ts (or a new *.test.ts).
Run npm test and npm run typecheck — both must pass.
Open a PR with a clear description and a sample of the output dataset.

Code & more: https://www.dailybuild.xyz/project/233-selfplay-factory

Self-Play Data Factory

Table of contents

Why this exists

Features

Architecture

The data flow

Technologies

Project layout

Quickstart

Point it at a real model (DeepSeek v4 flash)

CLI reference

The verifier algorithm — the heart of it

1. Letter scale A–T (20 letters), not digits

2. Logprob extraction

3. Expectation over the distribution

4. Criteria decomposition

5. Repeated evaluation (optional)

Fallbacks

The factory pipeline

DPO dataset format

The mock LLM

Unit tests

Forking & contributing

Get set up

Where to look first

Suggested new features (good first contributions)

Contribution workflow

Tags

Author

Stats

Published

You Might Also Like

What Do You Do While AI Codes?

Your AI Remembers Everything and Trusts All of It

Nobody Argued For Your Stack

Velocidade de entrega e custo de manutenção pós IA

I Tested 5 Design to Code Tools With the Same Outdated SaaS Dashboard

I Asked for a Portfolio but Got a Filing Cabinet