We built a real-time writing coach that scores tone and clarity as you type — think "this sentence reads as passive-aggressive" nudges inside a feedback composer. The pitch from day one was privacy: never send a user's draft performance review to a server. So we tried to run the whole thing client-side with WebGPU. It mostly worked. Here's the actual bill.

🎯 The Setup

We landed on a 3.8B parameter instruction-tuned model (Phi-3-mini class), quantized to 4-bit, served through web-llm so we got a WebGPU-accelerated inference loop without writing our own compute shaders. The quantization step looks like this if you're prepping your own checkpoint:

bash

convert + quantize with mlc_llm

mlc_llm convert_weight ./phi-3-mini-hf \
--quantization q4f16_1 \
-o ./dist/phi-3-mini-q4f16

mlc_llm gen_config ./phi-3-mini-hf \
--quantization q4f16_1 \
--conv-template phi-3 \
-o ./dist/phi-3-mini-q4f16

q4f16_1 gets you weights at roughly 2.2GB instead of the original ~7.6GB in fp16. That number matters more than almost anything else in this post.

⏱️ Load Time: The First Wall

A cold load means fetching ~2.2GB of weight shards, compiling WebGPU pipelines, and warming up KV-cache buffers. On a decent laptop over home wifi, that's 25–40 seconds before the first token. On a phone over LTE, we saw 90+ seconds, and a chunk of users just bailed.

The fix isn't clever inference — it's aggressive caching:

javascript
import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine("Phi-3-mini-4k-instruct-q4f16_1-MLC", {
initProgressCallback: (p) => updateLoadingBar(p.progress),
appConfig: {
useIndexedDBCache: true, // persist shards across sessions
},
});

With IndexedDB caching, repeat visits drop to 2–4 seconds for cache validation plus pipeline recompilation (WebGPU doesn't let you cache compiled shaders across sessions yet, so you pay that tax every load). First-visit latency stays brutal no matter what. We ended up pre-warming the model during onboarding — while the user reads a privacy explainer, the weights download in the background. It's a UX trick, not a technical fix, and I want to be honest about that distinction.

🧠 Memory Ceilings

This is where WebGPU stopped feeling like a drop-in replacement for a server call. Browsers cap maxBufferSize and maxStorageBufferBindingSize per adapter, and those caps vary wildly by GPU and OS. You can check them before committing to a model size:

javascript
const adapter = await navigator.gpu.requestAdapter();
const limits = adapter.limits;
console.log(limits.maxBufferSize, limits.maxStorageBufferBindingSize);

if (limits.maxStorageBufferBindingSize < 1_073_741_824) {
// under 1GB binding — fall back to a smaller model or WASM
loadFallbackModel();
}

In practice: integrated GPUs on mid-range Windows laptops and most Android devices choked on the 3.8B model once you factored in KV-cache growth during longer conversations. We had to drop to a 1.1B model (TinyLlama-class) for any device reporting navigator.deviceMemory < 8 or failing the buffer check above. Quality dropped noticeably — shorter effective context, more generic phrasing, occasional repetition loops after 6–8 turns.

iOS Safari was its own category of pain. WebGPU landed there only recently and still has rough edges with larger buffer allocations; we gate it entirely and fall back to WASM, which is 5–10x slower per token but at least doesn't crash the tab.

⚖️ Where the Server Still Wins

I want to push back on the breathless "browser LLMs eliminate the need for servers" narrative, because for this product it's not true.

• Quality ceiling. A quantized 3.8B model is simply worse at nuanced tone analysis than a hosted 70B+ model. For straightforward "this sounds harsh" feedback it's fine. For "rewrite this to sound confident but not dismissive in a cross-cultural context," it isn't.
• Consistent latency. Server inference on fixed hardware gives you a predictable p95. Client-side inference on a user's five-year-old Chromebook gives you a coin flip.
• Battery and thermal cost. Sustained WebGPU inference pulls meaningfully more power than a network request. On laptops this is a minor annoyance; on phones, users notice their device heating up during a ten-minute coaching session.
• Update velocity. Shipping a model fix server-side is a deploy. Shipping it client-side means another 2.2GB download for everyone, which we're not going to push more than once a quarter.

So we ended up hybrid, which is the less exciting but more honest answer:

javascript
async function getFeedback(text) {
if (canRunLocalModel() && text.length < 400) {
// fast, private, good enough for in-the-moment nudges
return await localEngine.chat.completions.create({
messages: [{ role: "user", content: buildPrompt(text) }],
max_tokens: 80,
});
}
// longer drafts or low-capability devices escalate to the server,
// with explicit user consent shown inline
return await fetch("/api/coach", {
method: "POST",
body: JSON.stringify({ text }),
}).then((r) => r.json());
}

The local model handles the cheap, frequent case — live micro-feedback as you type a sentence. The server handles the expensive, infrequent case — "give me a full rewrite of this paragraph." Users get the privacy win for the thing that actually feels invasive (constant monitoring of every keystroke) and the quality win for the thing they only ask for occasionally.

🚀 What Actually Shipped

• Device capability check on load (WebGPU support, buffer limits, deviceMemory) decides model tier: 3.8B / 1.1B / server-only.
• IndexedDB caching of weight shards, cut repeat load time by ~90%.
• Hard cutoff on KV-cache growth — we reset context every 10 turns rather than let memory creep until the tab crashes.
• Explicit UI distinction between "processed on your device" and "sent to our servers" so the privacy trade-off is visible, not assumed.

The local model isn't a server replacement — it's a different tool with different failure modes, and pretending otherwise just moves the disappointment to production.

If you've shipped WebGPU inference for something real: what was your actual memory ceiling in practice, and did you bucket devices by capability or just pick one model size and eat the failures? I'd like to know if our 8GB deviceMemory cutoff was too conservative or not conservative enough.