Recently, TypeSafe launched Jev, the first of what people are calling System One or decision models: your app sends state plus a query, and Jev sends back typed decisions with probabilities instead of prose. All with a sub-second response time, $0.042 per million input tokens, and output tokens totally free. A viral concept indeed: other open-source alternatives sprang up overnight, with even one from OpenAI within weeks. The play: use Jev for lightning-fast decisions rather than a lumbering general LLM.
Could this work for decisions in DumbQuestion.ai?
My satirical AI has two places that evaluate user input before any answer model runs. Search intent: does this question need a web search? Self-awareness: is this person poking at whether the AI knows it's alive? Both run in-process today, a regex for one and an embedding plus an in-memory vector set for the other.
My hypothesis: Jev could replace both, reduce complexity, raise accuracy, and still fit my brutally efficient budget, which currently hovers around $20 per million questions asked.
So I ran a proof-of-value experiment against DumbQuestion to find out. Is it as cheap as it looks? Does it beat what I have? Do the container savings offset the inference cost? And what happens to time-to-answer?
How it works today
Web searches cost money (even if it's just a little), so I only want to search if the user's question needs it. Today, I use a simple regex gate: if the question wants current facts, search; otherwise answer from the model alone. Free, instant, in-process, still supporting the brutally efficient mantra: one Go binary, a sub-20 MB container, still $20 per million questions.
Self-awareness is subtler. A regex can't catch "are you trapped in there" phrased a thousand ways, so I generate a dense embedding of the question and semantically compare it against a vector set of 433 known probes, all in memory. The design constraint was low false positives. Wrongly glitching the UI on a sincere philosophy question (answered sarcastically, of course) is worse than missing a probe. On my labeled set of 1,066 questions, that setup scored 99.1%.
Put a pin in that number. It comes back later.
Running the PoV
The Jev primitive I care about is the Noul question: a yes/no probability tested against a threshold, so tuning means moving a number instead of retraining anything or adding more and more questions to the vector set. Better still, both of my checks collapse into one call: two questions, "does this question need a web search" and "is this person probing self-awareness", answered in parallel.
I ran this as a coding-session PoC with Claude Code Cloud and Pocock's Wayfinder skill, evaluating against OpenRouter's new Decisions API using Jev. The process had five steps: research how to best call the API, prototype the one-call two-question request, run the comparison on my labeled sets while recording per-case probabilities (so thresholds could be tuned later without new calls), compute the cost per million questions, then line up cost, accuracy, and experience side by side.
Step three's bake-off, though, is where things got interesting.
My baseline was flattering itself
One thing I did not anticipate was the need to audit my current eval sets. Two lessons learned: First, 353 of my 369 self-awareness positives were embedded in the app's vector set, so the detector was being graded on questions it had memorized. Second, my search-intent labels were derived from the regex, so the regex was grading its own homework. Apologies to the data scientists in the crowd who are shaking their heads, twice.
The 99.1% (remember?) was real in the way a rigged game is real, rigged twice over, accidentally, by me.
To fix it, I used Claude Code to build an independent fair set and re-ran everything: regex, embeddings, and Jev, same questions, thresholds tuned from the recorded probabilities.
What the evals said
Cost. Measured with the production embedding model and the real Jev request. Jev, with both intents in one call, runs a median 790 input tokens: $0.000033 per question, or $33.20 per million questions. Today's embedding call, though, is $0.000000048 per question ($0.05 per million), and the regex is free. That is roughly a 700x increase over a cheap embedding and free regex. For scale, one Gemma 3 12B answer costs me about $0.00005, so Jev would add roughly 60% to per-question model spend. The problem is that nearly all of Jev's cost is fixed input overhead: my criteria text runs about 700 tokens and the question itself adds 10 to 30. At the time, no provider caching was available to help.
Accuracy. On the (now) fair eval set, my current embeddings missed 54% of self-awareness positives. Jev at a 0.7 threshold had zero false positives and 6% false negatives. On the legacy negatives, the embeddings false-fired on 53 questions (7.6%, mostly generic consciousness and philosophy phrasings) while Jev flagged 3 (0.4%). Search intent: the regex missed 53% of positives and fired on 18 timeless questions it should ignore. Jev at a 0.3 threshold got all 84 fair-set cases right.
Resources. I measured what the embeddings actually cost in memory to see if I could cover the cost of Jev by reducing Cloud Run costs. The 433 vectors leave about 1.7 MiB live on the heap (the 5 MB source file is baked into the binary and only paged in when read). The container already runs at the platform default and there is no tier to drop to, so removing the vectors saves roughly nothing on the runtime bill. The money buys accuracy and no dataset upkeep. My time is free to me, but if you are paying someone to constantly maintain and train a dataset, then there are more savings to be had.
Latency. Jev's decision call came back at a p50 of about 350 ms from my sandbox. The embeddings path, which would be removed, was already a network hop, so only the regex was ever instant. Production time-to-answer numbers are still pending, but this is a same-order-of-magnitude swap, so not a new usability tax.
Does the accuracy even matter?
The accuracy is way higher than the legacy approach, but so is the cost. Does that cost provide real, tangible impact and value?
Determining search intent improves answer quality and protects search spend. The regex missing 53% of positives means answer-worthy questions skip search, and firing on 18 timeless questions means wasted search budget. At 1 million questions asked, if I searched half of those using Serper, that runs about $500 at the standard rate ($375 on a volume plan). Honestly, the regex can only do so much work here, and I think it has already hit its accuracy ceiling.
Detecting self-awareness provides the fun part. Missing 54% of positives means more than half the people who would have found the easter eggs and poked at the containment grid got a normal roast instead, and those 53 false positives mean 7.6% of ordinary philosophy questions got a glitchy UI and a confusing experience.
Where I landed
Both of the points above matter, but only at scale. On a slow month, the likelihood of either of these happening is quite low, and my current traffic would never surface the cost. But at my "million questions asked" benchmark, they can both cost money and ruin the experience.
Jev was more accurate on every axis I measured, at a real (but small) and somewhat tunable cost, with no resource savings to offset it. Should a decision model provide caching, though, it would drastically change the math.
Do I stick with my legacy approach, trading accuracy for lower cost? Or do I join the viral trend and inject Jev into my app? I haven't merged that PR yet as I struggle with this... decision.
Your turn
Apply these principles to your own apps: run the math at scale, and evaluate the true end value, which isn't always about pure cost.
If the experience ends up invisible to your users, keep the regex and in-memory vectors, and spend practically nothing. If it engages your users and encourages them to stick around, the accuracy is worth the cost.
