Reef Review: Open-Source Continual Learning for AI Agents

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Reef is open-source infrastructure for agents that keep learning after deployment. It is built by Human-Agent-Society — a research collective anchored at MIT with contributors from UW, Meta, and Amazon AGI — and released under Apache-2.0. Three weeks after its public launch on September 15, 2026, the repo sits at 4,577 stars, 391 forks, 30 contributors, and shipped v0.1.0 on September 23. Key facts:

• Serve, observe, grow, commit: Reef is an OpenAI/Anthropic-compatible inference server that records every request, attaches feedback to it, runs a learning "recipe" asynchronously, and publishes accepted updates as versioned releases — without restarting serving.
• Evolves weights and harness: model weights via Slime + SGLang (NCCL hot-swap), or prompts/rules/skills via the Cordis harness-evolution backend. Harness recipes need only a model endpoint, no GPUs.
• Every response carries a receipt (x-reef-agent-record-id); your app posts a score or free-text feedback against it later.
• Release gating: candidates are evaluated before publication; rejected ones leave the current version serving. Artifacts live in Git LFS with an append-only, compare-and-swap release chain.
• Recipes with measured results: OpenClaw-RL (learns user taste from ordinary chat), SkillClaw (+12.05 points on WildClawBench Productivity), SAO, GEPA, TTT-Discover, Reefine.
• Reality check: v0.1.0, Python 3.12+, git-lfs required, weight training needs a 4–7 GPU stack, and the code is moving fast (56 commits in the last week).

If you have ever wished a production agent would learn from being corrected instead of repeating the mistake tomorrow, Reef is the most complete open-source answer available — and the most honest about how much infrastructure that takes.

Quick Reference

What Reef Is

The launch post's pitch is "your inference server is secretly a learner." The standard LLM lifecycle — train, evaluate, deploy, serve — assumes inference is the end of the pipeline. For agents it isn't: every session produces trajectories, tool results, and user reactions, and today most of that is discarded the moment the response is returned.

Reef inverts that. It is an inference server first (fronting SGLang, vLLM, or any upstream API), but every interaction becomes a record in a structured experience stream. Feedback attaches to records; learning recipes consume eligible records and produce candidate updates; an evaluation gate decides whether a candidate replaces what's serving. The loop runs asynchronously and users never see a restart.

The second half of the thesis: the model isn't the only thing that should evolve. Many production failures blamed on "the model" are harness bugs — a tool called with wrong arguments, memory retrieving the wrong context, a prompt inviting a plausible answer where a verified one was needed. Retraining weights for those is expensive and often unnecessary, so Reef treats the harness (prompts, rules, skills, tool wiring) as a versioned artifact evolved from experience, exactly like weights.

The README's positioning table shows where it sits:

RL frameworks bundle an inference engine to generate training data, not to serve applications. Reef starts from serving and builds learning around it.

Why It's Trending Now

Three things converged in September 2026. "Recursive self-improvement" became a roadmap item at every lab while open source had no shared substrate to experiment with; Reef's launch post frames itself as landing "before all the RSI hype becomes real." The research it packages already had traction: OpenClaw-RL (arXiv:2603.10165) trains a personal agent purely from what the user did next — moving on counts as acceptance, a complaint as rejection — and SkillClaw (arXiv:2604.08377) evolves skills nightly from a day's sessions. Both were papers with bespoke code; Reef turns them into recipes on one runtime alongside SAO, GEPA, and TTT-Discover.

And the team is credible and shipping fast. Ao Qu is a PhD student at MIT's Institute for Data, Systems, and Society; Bo Liu worked on self-improvement and self-play at Meta FAIR; Han Zheng did agentic post-training at Amazon AGI. Since the August 31 repo creation there have been ~470 issues and PRs, v0.0.2 on September 2, v0.1.0 on September 23, and a full docs site with recipe result pages.

The Core Loop: Serve → Observe → Grow → Commit

Reef processes every learning cycle in four steps, each mapped to a package in the repo:

The unit of organization is a scenario, named by the x-reef-scenario request header. Each scenario has its own append-only release chain of artifacts — model checkpoint, LoRA adapter, harness tree, or routing policy — advanced with compare-and-swap so a stale publisher can't overwrite a newer release. Weights are stored through Git LFS, which is why git-lfs is a hard dependency even for the PyPI install.

Getting Started: Serve, Score, Learn

Installation is standard uv:

git lfs install
uv venv && source .venv/bin/activate
uv pip install reef-infra
python3 -c "import reef; print(reef.__version__)"   # 0.1.0

The minimal deployment is Reef as a pure inference server in front of a local model:

uv run reef serve --inference.model-path Qwen/Qwen2.5-1.5B-Instruct

The interesting part is the client side. Reef's endpoints take the provider's own request body, and the response gains one header — the receipt:

import os
import httpx

reef = httpx.Client(
    base_url="http://127.0.0.1:8900",
    headers={
        "Authorization": f"Bearer {os.environ['REEF_TOKEN']}",
        "x-reef-scenario": "hello-reef",
    },
    timeout=300,
)

response = reef.post(
    "/v1/chat/completions",
    json={
        "model": os.environ["MODEL_PATH"],
        "messages": [{"role": "user", "content": "Return exactly: reef is ready"}],
    },
)
response.raise_for_status()

receipt = response.headers["x-reef-agent-record-id"]
answer = response.json()["choices"][0]["message"]["content"]

matched = answer.strip() == "reef is ready"
reef.post(
    "/reef/report",
    json={
        "score": float(matched),
        "feedback": "matched" if matched else "wrong answer",
        "references": [receipt],
    },
).raise_for_status()

A report carries a numeric score, free-text or structured feedback, and the receipts it evaluates; the schema in reef/core/reports/ is validated at the endpoint. Once a recipe has enough eligible feedback it runs a training step and syncs new weights into the serving engine via NCCL. Later requests use the new version; nothing restarts.

Recipes are selected in the serve config, not by the request:

# serve.yaml
reef:
  recipe: recipes.sao.recipe:SAORecipe
  batch_size: 1
  max_staleness: 18

reef serve -c recipes/sao/examples/sao/serve.yaml

max_staleness is a tell that the team has actually run this: training is asynchronous with serving, so records go off-policy, and the recipe bounds how stale a rollout may be before it's excluded.

Harness Evolution Without GPUs

The path most readers can try tonight is Reefine, the built-in harness-refinement recipe. It needs a model endpoint — Ollama is fine — and no training GPUs:

reef serve --recipe reefine \
  --inference.upstream-url http://127.0.0.1:11434 \
  --inference.upstream-model gemma4:26b

Then create a scenario, install the Reef-wrapped harness (currently a pi coding-agent adapter), and file a plain-language ask:

curl -fsS -H "Content-Type: application/json" \
  -d '{"name": "my-harness"}' http://127.0.0.1:8901/reef/scenarios

curl -fsS -H "x-reef-scenario: my-harness" \
  'http://127.0.0.1:8901/reef/harness/install?adapter=pi' | bash

reef-pi evolve "when I ask you to fix a bug, reproduce it with a failing test first"

The served model writes that ask as a skill, a rules entry, an agent command, or a pi extension. Where the host can isolate it (Linux with bwrap and pasta as non-root), Reef runs the changed harness as a coding agent before handing the change back. The next session shows an update notice; /versions lists releases and /versions <n> install pins one. A Claude Code adapter landed in PR #118.

This is the piece that made me sit up. Skills in the OpenClaw / Claude Code sense are text files humans maintain by hand. Reef makes them a release artifact produced from usage, evaluated, and versioned — the lifecycle discipline you'd apply to weights.

The Recipe Catalog and What's Actually Measured

The recipe table is organized by task type × what evolves:

Two results pages are worth reading in full.

OpenClaw-RL on Reef (August 27, 2026): a stock hermes-agent answers 72 GSM8K homework problems for a simulated Qwen3-32B "student" who wants solutions that don't look AI-written — no bold, no lists, every step shown, correct answer. The student never states this; the agent must infer it from reactions. Policy and PRM are both Qwen3-4B-Thinking-2507 on a seven-GPU layout. Bold and list rates fall session over session, and the run hits the paper's adaptation criterion (three passes in a row) at session 14. Hermes "knows nothing about Reef" — a header shim injects the scenario tag and training happens underneath.

SkillClaw on Reef (preliminary, August 29, 2026): GLM-5.3-Flash on 4 local GPUs, six nights of evolution applied 13 skill improvements and 8 creations, growing the pool from 9 to 17. In the Productivity category the final day beat a frozen control by +12.05 points (2.29 sd). The gain criterion was preregistered, and the README labels the result preliminary.

Every result page documents task, setup, measurements, and limitations, with python run.py report to recompute — more rigor than most agent projects at 4k stars bother with.

Architecture Notes for Operators

• Weight training is a real GPU deployment. The Slime backend needs Ray, a Slime driver, and CUDA builds of torch, SGLang, and Megatron; you build the Docker image and run it with --gpus all --network host --ipc host --shm-size 32g.
• Tinker path for CPU-only hosts. "Train with Tinker" runs hosted LoRA training from a GPU-less machine — the middle ground between harness-only and a seven-GPU rig.
• vLLM is arriving. PR #571 added token-native capture and a vLLM runtime kind; SGLang has been the serving engine so far.
• State lives on disk (.reef/reefine/, /var/lib/reef) and every accepted version is retained — plan for growth.
• Auth is a bearer token (REEF_TOKEN); Reefine listens unauthenticated on 127.0.0.1:8901 by default. Don't expose it.
• v0.1.0 added evolution budgets and a whole-tree credential scan (PR #146) — an LLM is writing files into your agent's config tree, so a gate that refuses to publish a leaked key matters.

Community Reaction

Reef hasn't had a Hacker News moment — no front-page thread as of September 24, and a thin Reddit footprint. Growth came from the Hugging Face launch post, X, Trendshift, and Chinese developer channels (README.zh.md, a WeChat group that's already full). RuntimeWire's coverage put it well: the release gate "carries much of Reef's practical value," because continual learning "can degrade a working system as easily as it can improve one."

The signal I weight more is the issue tracker. Open RFCs are substantive: adaptive KL for online policy updates (#466), SFT cold-start without a resident inference engine (#586), named release components so a scenario evolves weights and harness together (#537), a self-distillation roadmap validated on CEO-Bench (#502), and reproducible continual-learning benchmarks (#357). It's a research group using GitHub as a lab notebook — good if you want to build on it, a warning if you want it to sit still.

Honest Limitations

• It is v0.1.0. Config names are already churning (--recipe harness-evolve became reefine). Expect breaking changes through the fall.
• Only Reefine ships in the wheel. SAO, OpenClaw-RL, SkillClaw, GEPA, and TTT-Discover live in the repo's recipes/ cookbook, loaded by dotted class reference from a source checkout.
• The hard part is still the reward. Reef is plumbing for feedback; it doesn't tell you what to score. OpenClaw-RL needs a separately deployed PRM, SkillClaw a benchmark grader, SAO a verifier. If your only signal is "user didn't complain," design that first.
• GPU appetite. The reference OpenClaw-RL layout is seven GPUs for a 4B model — a research budget, not a side project.
• Sandbox caveat. The harness proposer edits and executes your harness. Isolation is Linux-only (bwrap + pasta); on macOS you opt into REEF_PROPOSER_SANDBOX=none.
• One polished harness adapter (pi), Claude Code in progress. No OpenClaw or Codex adapter yet, despite the OpenClaw-RL naming.

Who Should Use Reef

Good fit: teams running an agent with a measurable outcome signal (test pass rates, verifier results, task completion) who want it to improve the agent automatically; researchers comparing continual-learning methods on one runtime instead of five paper codebases; anyone hand-maintaining a coding-agent skill library who wants to try evolving it from sessions (Reefine is the low-risk on-ramp).

Poor fit: teams that need memory, not learning — "remember the user" is far cheaper with a memory layer like Honcho or Mem0; anyone without a reward, since Reef amplifies whatever signal you give it, including noise; production deployments that need API stability this quarter.

Comparison with Alternatives

RL frameworks are for training runs, prompt optimizers for offline compilation, memory layers for recall. Reef is the first open project that treats all three as recipes over one live serving stream with a release pipeline in front — GEPA is literally one of its recipes.

FAQ

Does Reef require GPUs?
Not for harness evolution. Reefine, SkillClaw, GEPA, and Meta-Harness need only a model endpoint (Ollama, OpenRouter, any OpenAI-compatible API). Weight recipes (SAO, OpenClaw-RL, TTT-Discover) need the Slime + SGLang GPU stack, or you can use the Tinker path for hosted LoRA training from a CPU machine.

Is Reef an inference engine like vLLM?
No — it sits in front of one. Reef exposes /v1/chat/completions and /v1/messages, forwards to SGLang (or an upstream API, with vLLM support landing), records every exchange, and hot-swaps accepted weight updates into the engine with NCCL. Your app talks to Reef the same way it talks to OpenAI, plus one header.

What is a Reef "recipe"?
A Python class (recipes.sao.recipe:SAORecipe, etc.) that defines how recorded interactions and feedback are processed, which learning algorithm runs, and when a candidate is evaluated and published. You pick one in serve.yaml; requests don't choose recipes.

How does Reef prevent a bad update from breaking production?
Every candidate goes through the recipe's evaluation gate and selection policy before it is published. Rejected candidates leave the current release serving. Accepted ones become a new, auditable version in an append-only chain that you can roll back to. v0.1.0 also added a credential scan across the whole harness tree before release.

Can I use Reef with Claude Code or OpenClaw?
The harness-install endpoint ships a pi adapter, and a Claude Code adapter was merged in PR #118. OpenClaw-RL is a method (learn from what the user does next), not an OpenClaw plugin — the reference experiment uses hermes-agent behind a header shim. Any agent that lets you set its base URL can route inference through Reef today; harness evolution needs an adapter.

What license is Reef?
Apache-2.0, including the recipes. Dependencies (SGLang, Slime, Cordis) and example datasets carry their own licenses.

Verdict

Reef is the most serious open-source attempt yet to make "the agent learns from its work" a systems property rather than a research demo. The receipt-and-report API can be added to an existing app in an afternoon, the release gate is the right safety primitive, and the results pages show real learning curves with honest caveats. What it is not is turnkey: weight training is a multi-GPU deployment, the API is pre-1.0, and you still supply the reward. Try Reefine against Ollama this week; budget a quarter before routing production traffic through a weight recipe. Reef turns your inference server into a versioned, gated learner — and 4.6k stars in three weeks says a lot of people wanted exactly that.

Sources

• Human-Agent-Society/reef on GitHub — README, recipes, v0.1.0 release notes (September 23, 2026)
• Your Inference Server is Secretly a Learner — Reef launch post, Hugging Face (September 15, 2026)
• Reef documentation — reefinfra.ai — quickstart, evolve-your-model GPU requirements, recipe catalog
• OpenClaw-RL on Reef — results, August 27, 2026 and SkillClaw reproduction — August 29, 2026 results
• RuntimeWire: Ao Qu and collaborators open-source Reef (September 15, 2026)