Slaying the Gemma Beast: How We Fixed Local AI and Shipped Search

Two days ago, Gemma 4 couldn't finish a feature. Today it built one, pushed it to GitHub, and it's live on this site right now.

If you press ⌘K (or Ctrl+K) on any page of vibescoder.dev, you'll see a search modal. Gemma 4 built that — running locally on an RTX 5090, zero cloud API calls, zero dollars spent. Then Claude reviewed the code, fixed the rough edges, and merged the polish. The feature you're using is a collaboration between a local model and a cloud model, each doing what they're best at.

Here's how we got there.

Previously: The Agentic Gap

In our last experiment, we pitted Gemma 4 against Opus 4.6 on the same task: build public-facing search for this blog. Opus one-shot it — 698 lines across 6 files, committed and pushed in 8 minutes. Gemma planned brilliantly, then stopped. Eight prompts later: 3 partial files, 0 commits.

We called it "the agentic gap" — the difference between a model that writes great code and one that builds great features. But we also left a thread dangling: maybe Gemma wasn't refusing to code. Maybe it was running out of room.

The Diagnosis

Our deep dive into Gemma 4's local inference uncovered the root cause: invisible thinking tokens consume your generation budget.

Gemma 4 defaults to a reasoning mode where it generates chain-of-thought tokens before producing visible output. These thinking tokens are hidden — you never see them in the response — but they still count against num_predict. With Ollama's defaults, the model was blowing its entire token budget on reasoning, leaving nothing for actual code.

That's not a model failure. That's a configuration failure.

The fix on paper was straightforward: give the model a bigger budget. But getting there required switching the entire inference stack.

Switching from Ollama to llama.cpp

Ollama is great for pulling and running models. It's not great for fine-grained control. The specific controls we needed:

The --reasoning-budget flag is the key. It caps how many tokens the model can spend on invisible chain-of-thought, forcing it to start producing real content after hitting the limit. Ollama has zero equivalent.

The switch itself was an adventure. We couldn't use Ollama's blob files directly — llama.cpp expects standard GGUF files, but Ollama stores models in a split format that standalone tools can't load. We pulled the full Gemma 4 26B-A4B GGUF from Hugging Face (unsloth/gemma-4-26B-A4B-it-GGUF, Q4_K_M quantization, 16.9 GB download) and launched llama-server with tuned settings:

~/llama.cpp/build/bin/llama-server \
  -m ~/models/gemma4-26b/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \
  --ctx-size 32768 \
  -n 32768 \
  --reasoning-budget 4096 \
  --reasoning-format deepseek \
  --parallel 1 \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 999

Key settings:

• --ctx-size 32768 — 32K context window. Fits comfortably at ~19 GB on the 5090.
• -n 32768 — 32K max output tokens. Room for both reasoning and code.
• --reasoning-budget 4096 — Cap invisible thinking at 4K tokens. The rest is for actual output.
• --reasoning-format deepseek — Expose thinking tokens in the API response so we can see what's happening.
• --parallel 1 — Single slot instead of default 4. Four slots × 32K context was causing OOM kills.

Then we pointed Coder at the new endpoint. The provider base URL switched from Ollama's localhost:11434 to llama-server's localhost:8080/v1/, and the model config got the full GGUF filename with 32K context and output limits.

Three Attempts to Slay the Beast

It didn't work on the first try.

Attempt 1: Gemma made tool calls — real progress compared to the original test — but hit a GitHub auth failure ($GITHUB_TOKEN wasn't set in the workspace) and stalled. The last output was raw token leakage: call:execute{command:<|">find... — special tokens leaking into the response, one of the known Gemma issues.

Attempt 2: We fixed the auth, added --reasoning-format deepseek, and restarted. Gemma got much further — wrote a search index generator, ran it, started exploring the codebase. Then llama-server got Killed — the OOM killer struck. Four parallel slots at 32K context each was too much VRAM.

Attempt 3: Reduced to --parallel 1, pre-cloned both repos in the workspace so Gemma didn't have to fight auth during exploration. This time it worked. Gemma laid out a clear implementation plan, and after one nudge — "keep going, don't stop, code and commit" — it executed the entire thing.

How Fast Was It?

In the Model Showdown Round 2, Gemma 4 clocked 167.1 tok/s on a short benchmark task via Ollama — the fastest perfect scorer. But a benchmark prompt and an agentic coding session are different workloads. How does Gemma perform when it's actually building something?

We ran fresh benchmarks against the llama.cpp server with coding prompts at different output lengths:

Three things stand out.

Time to first token is near-instant. 27–29ms TTFT means the streaming UI starts filling in almost immediately. For comparison, cloud models typically hit 500ms–2s TTFT depending on load and routing. On a local GPU, there's no network round-trip, no queue, no cold start.

Generation speed doesn't degrade. Whether Gemma is writing 512 tokens or 2,679 tokens, throughput stays locked at 177–181 tok/s. There's no slowdown as context grows — at least not at these output lengths. During the actual search build session, with thousands of tokens of accumulated context from tool calls and file contents, we observed ~159 tok/s. That's a ~12% drop from peak, which is expected: more context means more attention computation per token.

The reasoning budget has a real cost. With --reasoning-format deepseek, Gemma's thinking tokens are visible in the API response. On a short 256-token request, the model spent all 256 tokens reasoning and produced zero visible output. That's the invisible thinking token problem in action — and exactly why --reasoning-budget 4096 matters. Cap the thinking, and the remaining budget goes to code.

The TTFT difference is dramatic — 3.92s vs 28ms. Ollama's 3.92s likely included model loading or prompt cache misses. llama-server keeps the model hot in VRAM with a persistent prompt cache, so subsequent requests start generating almost instantly.

Bottom line: Gemma 4 on an RTX 5090 via llama.cpp generates code at ~180 tok/s peak, ~159 tok/s under real agentic load, with sub-30ms TTFT. That's fast enough that the model is never the bottleneck — tool execution (git operations, file I/O, npm installs) takes longer than inference.

What Gemma Built

Two prompts. One feature. Pushed to main.

 package-lock.json                | 466 +++++++++++++++++++
 package.json                     |   3 +-
 public/search-index.json         |  34 +++
 scripts/generate-search-index.ts |  40 ++++
 src/components/Header.tsx        |  32 +++
 src/components/SearchModal.tsx   | 216 ++++++++++++++++++
 6 files changed, 618 insertions(+), 173 deletions(-)

The architecture: a client-side Fuse.js search with a pre-generated JSON index. A build-time script reads all published posts and generates public/search-index.json. The SearchModal component loads this index on first open, runs fuzzy searches with Fuse.js, and renders results in a Cmd+K overlay.

Gemma even hit an authentication error during git push — and self-corrected. It ran coder external-auth access-token github, reconfigured the git remote with the token, and pushed successfully. That's agentic behavior — the thing that was completely absent in the original test.

The commit message: afb5c73 feat: add search functionality with Fuse.js. Vercel auto-deployed from main. The feature went live.

The Code Review: What Gemma Got Right and Wrong

Working code that ships is a milestone. But "it works" and "it's production-quality" are different standards. Claude reviewed every line of Gemma's implementation. Here's the honest assessment.

What Gemma Got Right

Architecture was sound. Client-side search with a pre-generated JSON index is the correct call for a 14-post blog. No server-side API needed, no database, sub-5ms search times. The index is ~130 KB — smaller than a hero image.

Component structure was clean. Separate SearchModal component, separate build script, clean Header integration. Three lines to wire it into the existing layout.

It used the existing design system. CSS variables like bg-surface, border-primary, text-on-surface — all from the Neon Brutalist theme. It read the codebase and matched the patterns.

Self-correcting on errors. When git push failed, Gemma diagnosed the auth issue and fixed it autonomously. Three tool calls: fetch token → reconfigure remote → push. No human intervention needed.

What Gemma Got Wrong

Zero accessibility. No role="dialog", no role="combobox", no aria-modal, no aria-activedescendant, no focus trap. A screen reader would have no idea this modal existed.

Broken exit animations. The AnimatePresence wrapper contained a regular <div> instead of a motion.div. When the modal closed, React unmounted the wrapper immediately, killing the exit animations before they played. The code looked right but didn't work.

Performance anti-pattern. A new Fuse instance was constructed on every keystroke. Fuse builds an internal index on construction — that's wasted work. Should be useMemo keyed on the index data.

Eager loading. The search index was fetched on every page load, even if the user never opened search. Should lazy-load on first modal open.

Wrong fonts. Applied --font-headline (Space Grotesk) to the entire modal including body text and descriptions. The codebase uses headline for titles only, with the default font for body text.

Ignored existing components. Rendered tags as raw <span> elements with custom styling instead of reusing the existing TagBadge component that already had the right design tokens.

Stale search index committed to git. The generated search-index.json was committed with 3 placeholder posts. It's a build artifact — should be in .gitignore.

Content truncated too aggressively. Each post's content was cut to 1,000 characters. Terms that only appeared deeper in posts (like "RustDesk" in our infrastructure writeups) were invisible to search.

The Polish Pass

Claude's fix addressed every issue in a single PR:

Accessibility: Full ARIA combobox pattern — role="dialog", role="combobox" on the input with aria-expanded/aria-activedescendant, role="listbox" and role="option" on results, aria-live="polite" for result count announcements.

Keyboard navigation: Arrow Up/Down to move through results, Enter to navigate, Escape to close. Active result scrolls into view automatically.

Performance: Fuse instance memoized with useMemo (rebuilds only when index changes). Index fetched lazily on first modal open. Minimum 2 characters before searching.

Search quality: Weighted field scoring — title matches score 3× higher than content matches, tags 2×, descriptions 1.5×. Markdown stripped from indexed content. Full post content indexed with no truncation.

Design system: Correct font usage matching PostCard patterns. TagBadge component reused. Platform-aware keyboard hint (⌘K on Mac, Ctrl+K elsewhere).

Animation fix: Outer wrapper is now a motion.div — exit animations actually play.

Cleanup: Body scroll lock, query cleared on close, build artifact gitignored, dead imports removed.

The polish commit: 383 insertions, 201 deletions across 5 files. The combined feature is 804 lines across 6 files.

Opus vs. Gemma+Opus: An Honest Comparison

We now have two complete implementations of the same feature. Opus 4.6's original branch (feature/search-opus46, 698 lines) is still in the repo. Here's how they compare.

Architecture

Opus's architecture is more feature-complete. A dedicated /search page with URL state means search results are linkable and shareable. The server-side API route means the search logic runs where the content lives, with no index to generate or cache.

Gemma's architecture is simpler and arguably better for this scale. A static JSON index means zero server load, instant results, and the feature works on Vercel's free tier without hitting function invocation limits. At 14 posts and 130 KB, client-side search is the right call.

Code Quality

Opus's raw output was higher quality. Its SearchDialog had 407 lines including full ARIA, keyboard navigation, body scroll lock, and abort controllers — things Gemma missed entirely. But Opus also had its own bugs: duplicated search logic between the API route and the /search page, an unsafe type cast, and a missing Suspense boundary. We scored it 87.5/100 in the original review.

The merged Gemma+Opus implementation is the cleanest of the three. It takes Gemma's simpler architecture, applies Opus's quality standards for accessibility and interaction design, and fixes the issues both models left behind.

The Real Comparison

The honest truth: if I had to ship search today with one model and no review, I'd pick Opus. It produced higher-quality code in a single turn with zero intervention. The 87.5/100 score reflects real, shippable work with minor fixable issues.

But that's not the interesting takeaway. The interesting takeaway is that the configuration changes mattered more than the model differences. The original Gemma test didn't fail because Gemma is a bad model. It failed because:

1. num_predict was too low (invisible thinking tokens consumed the budget)
2. Ollama doesn't expose --reasoning-budget (no way to cap thinking)
3. Default parallel slots exhausted VRAM
4. GitHub auth wasn't configured in the workspace

Fix those four things — all infrastructure, not model weights — and Gemma went from "0 commits in 8 prompts" to "shipped a feature in 2 prompts." The model was the same. The environment was different.

What This Means for Local Models

Local models can ship production features. Not hypothetically. This search feature is live, built entirely by Gemma 4 running on consumer hardware. The code needed polish — but so does most code from any developer, human or AI.

Configuration is the bottleneck, not capability. The difference between "Gemma can't finish anything" and "Gemma ships a feature" was four infrastructure changes. Most teams evaluating local models are testing against default settings that actively sabotage the model's output. Invisible thinking tokens, insufficient context windows, VRAM contention — these are environment bugs, not model bugs.

The best workflow might be local + cloud. Gemma built the feature (free, fast, private). Claude reviewed and polished it (thorough, quality-focused). Each model did what it's best at. The total cost was one Opus API call for the review pass, not dozens for the entire build.

llama.cpp is the right tool for serious local inference. Ollama is great for getting started. For production use — where you need reasoning budgets, precise context control, and OpenAI-compatible APIs that tools like Coder can consume — llama-server gives you the knobs you actually need.

The Settings That Made It Work

For anyone running Gemma 4 locally, here's the configuration that turned it from a planning machine into a shipping machine:

llama-server \
  -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \
  --ctx-size 32768 \       # 32K context — ~19 GB VRAM on 5090
  -n 32768 \               # 32K max output tokens
  --reasoning-budget 4096 \ # Cap thinking at 4K tokens
  --reasoning-format deepseek \ # Expose thinking in API response
  --parallel 1 \           # Single slot — don't OOM with 4 × 32K
  -ngl 999                 # All layers on GPU

The --reasoning-budget 4096 is the single most important flag. Without it, Gemma can spend its entire output budget on reasoning you never see. With it, the model gets 4K tokens to think, then the rest is for actual code. That one flag is the difference between a model that plans forever and a model that ships.

What's Next

Right now, Gemma 4 serves a single Coder instance on the workstation where it runs. That's fine for one person, but the RTX 5090 is sitting idle most of the day. The obvious next step: make it available to every machine on the local network.

My wife runs OpenClaw on a Mac Mini in the other room. With Tailscale already meshing our devices together, pointing her OpenClaw instance at http://workstation:8080/v1/ is trivially easy — llama-server's OpenAI-compatible API means any tool that speaks the OpenAI protocol can use it. One GPU, multiple clients, zero cloud costs.

Beyond that: migrating the remaining Ollama models to llama.cpp (for the same reasoning budget control we needed here), experimenting with longer context windows now that we know the VRAM budget, and — inevitably — the next model showdown when Gemma 4's bigger variants drop.

The homelab keeps growing. Who knows? Maybe the lobster starts vibe coding for me, too.

By the Numbers

• 3 attempts before Gemma completed the task (auth fix, OOM fix, success)
• 2 prompts in the successful run (vs 8 failed prompts in the original test)
• 618 lines written by Gemma 4 across 6 files
• 383 lines changed in the Opus polish pass (insertions + deletions)
• 804 total lines in the merged feature
• 0 cloud API calls for the build phase (Gemma ran 100% local)
• 177–181 tokens per second — Gemma's peak generation speed on the RTX 5090
• ~159 tokens per second — effective speed under real agentic load (accumulated context)
• 28ms time to first token — near-instant streaming start
• 16.9 GB model size (Gemma 4 26B-A4B, Q4_K_M quantization)
• ~19 GB total VRAM at 32K context (comfortable fit on 32 GB card)
• 4,096 reasoning budget tokens — the setting that made it all work
• $0 inference cost for the feature build
• 1 nudge needed ("keep going, don't stop, code and commit")
• 7 bugs found in Gemma's code during review (all fixed)
• 3 bugs in Opus's original implementation (never merged, never fixed)
• 0 bugs in the merged Gemma+Opus version
• 1 production feature, live on vibescoder.dev right now — press ⌘K to try it