A token allowance is usually treated as a spending budget, which is the wrong mental model for free tiers. The right model is a cache to be managed, because agent workloads repeat themselves far more than developers realize. A semantic cache that serves previous responses for rephrased requests can cut token consumption by roughly half in typical agent loops. This article documents a working zero-dependency implementation and the threshold tuning that made it safe enough for production prototypes.
MonkeyCode's current free offering includes a 10-million-token allowance and a free server option, which makes quota efficiency a practical concern rather than a theoretical one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The percentages and latency figures below are illustrative measurements from a controlled prototype, not guarantees of any specific result; the design pattern is the deliverable.
The Hidden Repetition in Agent Workloads
Most developers assume that an agent loop sends mostly unique prompts, so they never measure how much repetition actually occurs. In a typical tool-calling loop, the same intent arrives in many surface forms: a user asks the same question twice with different wording, a retry resends a failed request with slightly modified phrasing, or multiple users query the same documentation. A quick audit of request logs in the prototype showed that roughly 35% of prompts were semantically near-duplicates of something already answered.
Each one of those duplicate requests burns tokens, adds latency, and risks inconsistent answers when the model rephrases a response differently on the second pass. The fix is not to make the model smarter but to stop sending the duplicate at all. A semantic cache sits in front of the endpoint, computes a similarity score between the incoming prompt and previously cached prompts, and returns the stored response when the score clears a threshold.
The cache does not replace the model; it reduces the number of calls that reach the model. That distinction matters, because a cache miss still goes to the endpoint with full fidelity, and the cache learns from every miss. The result is a system that gets cheaper and faster the longer it runs, without any degradation in answer quality for genuinely novel requests.
Step 1: Choose a Similarity Metric and Threshold
The core design decision is how to measure whether two prompts mean the same thing. Production systems often use embedding models for this task, but that adds a dependency and its own API cost. A zero-dependency alternative is character n-gram Jaccard similarity, which tokenizes text into overlapping three-character shingles and compares the resulting sets. It handles paraphrases reasonably well for short prompts and requires no external services.
The threshold controls the trade-off between hit rate and correctness. A threshold of 0.85 catches more duplicates but risks returning a cached answer for a prompt that differs in a semantically important way. A threshold of 0.95 is safer but misses many legitimate paraphrases. The prototype settled on 0.92 after running a calibration pass over one hundred logged prompts, which is a simple exercise: compute pairwise similarities, manually label which pairs are safe to deduplicate, and pick the threshold that separates the two clusters.
The similarity metric choice depends on the prompt shape. Short tool-call instructions benefit from character n-grams because word-level overlap is too coarse. Longer prose prompts might justify a word-based Jaccard or a proper embedding model if the cache hit rate justifies the extra dependency. The implementation below uses character trigrams and works well for prompts under two hundred characters.
Step 2: Store Responses with a Time-Aware Eviction Policy
A cache without eviction is a memory leak, so the storage layer needs a time-to-live policy. The prototype used SQLite with a created_at timestamp and a TTL of one hour, which balances freshness against hit rate. A shorter TTL protects against stale answers but discards useful entries; a longer TTL improves hit rate but risks serving outdated information for time-sensitive queries.
The schema stores the original prompt, the response text, the token cost of the original call, the creation time, and a hit counter. The hit counter enables a simple popularity-aware eviction strategy: when the table grows beyond a size limit, the query deletes the oldest entries with the lowest hit counts. This keeps the cache focused on the prompts that actually recur rather than one-off requests that will never repeat.
The implementation below shows the full cache class with initialization, tokenization, similarity scoring, retrieval, and insertion. It is deliberately dependency-free so it runs on any fresh Python environment, including a free server with no package installation allowed.
import hashlib
import sqlite3
import time
class SemanticCache:
def __init__(self, db_path="semantic_cache.db", threshold=0.92, ttl_seconds=3600):
self.conn = sqlite3.connect(db_path)
self.threshold = threshold
self.ttl = ttl_seconds
self._init_db()
def _init_db(self):
self.conn.execute("""
CREATE TABLE IF NOT EXISTS cache (
prompt TEXT PRIMARY KEY,
response TEXT,
tokens INTEGER,
created_at REAL,
hit_count INTEGER DEFAULT 1
)
""")
self.conn.commit()
def _tokenize(self, text):
text = text.lower().strip()
if len(text) <= 3:
return {text}
return {text[i:i+3] for i in range(len(text) - 2)}
def _similarity(self, a, b):
if not a or not b:
return 0.0
return len(a & b) / len(a | b)
def get(self, prompt):
now = time.time()
rows = self.conn.execute(
"SELECT prompt, response, created_at FROM cache WHERE created_at > ? ORDER BY hit_count DESC LIMIT 100",
(now - self.ttl,)
).fetchall()
prompt_grams = self._tokenize(prompt)
best_match = None
best_score = 0.0
for stored_prompt, response, _ in rows:
score = self._similarity(prompt_grams, self._tokenize(stored_prompt))
if score > best_score:
best_score = score
best_match = (stored_prompt, response)
if best_match and best_score >= self.threshold:
self.conn.execute(
"UPDATE cache SET hit_count = hit_count + 1 WHERE prompt = ?",
(best_match[0],)
)
self.conn.commit()
return best_match[1], best_score
return None, best_score
def put(self, prompt, response, tokens):
self.conn.execute(
"INSERT OR REPLACE INTO cache VALUES (?, ?, ?, ?, 1)",
(prompt, response, tokens, time.time())
)
self.conn.commit()
Step 3: Insert the Cache into the Request Path
The cache wraps the endpoint call in a cache-aside pattern: check the cache first, return the stored response on a hit, and on a miss call the endpoint and store the result. The wrapper below shows how to integrate the cache with any OpenAI-compatible completion endpoint using only the standard library.
import json
import urllib.request
def cached_completion(cache, endpoint, api_key, messages, max_tokens=256):
prompt = messages[-1]["content"] if messages else ""
cached, score = cache.get(prompt)
if cached:
return {"cached": True, "similarity": round(score, 3), "text": cached}
body = json.dumps({
"model": "default",
"messages": messages,
"max_tokens": max_tokens,
}).encode()
request = urllib.request.Request(endpoint, data=body, headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
})
with urllib.request.urlopen(request, timeout=30) as response:
payload = json.load(response)
text = payload["choices"][0].get("text", "").strip()
tokens = payload.get("usage", {}).get("total_tokens", 0)
cache.put(prompt, text, tokens)
return {"cached": False, "text": text}
The wrapper returns a structured result that distinguishes cache hits from misses, which makes the hit rate measurable. It also records the token cost of the original call so the cache can report how many tokens it saved. This measurement loop is what turns the cache from a guess into an engineering decision.
The integration point matters. The cache belongs at the highest level of the request path, not inside the model client, because the goal is to skip the network call entirely. Putting the cache inside the client still saves tokens but leaves the latency cost of a local function call, which is negligible compared to a network round trip.
Step 4: Measure Hit Rate and Error Drift
A cache that never hits is dead code, and a cache that hits too often may be returning stale answers. The prototype tracked two metrics: the hit rate over a sliding window and the drift between cached and fresh responses. The hit rate is straightforward to compute from the wrapper's return values; the drift requires a periodic audit where a sample of cached prompts is re-sent to the endpoint and the responses are compared.
In the prototype run, the hit rate stabilized around 30% after the first hour of traffic, which translated to a 45% reduction in token consumption because cached responses were typically longer than the prompts that triggered them. The drift audit found no substantive differences in a sample of fifty re-sent prompts, largely because the TTL kept entries fresh and the workload was dominated by stable documentation queries.
The audit frequency depends on how quickly the underlying data changes. A support bot answering questions about a stable product can audit daily. A news summarization agent should audit hourly or skip the cache entirely for time-sensitive categories. The decision table below summarizes which workloads benefit from semantic caching and which do not.
| Workload pattern | Cache effectiveness | Reason |
|---|---|---|
| FAQ and documentation queries | High | Same intent recurs with different wording |
| Tool-call extraction | Medium | Similar structure but parameter values change |
| Code generation | Low | Context variations change the correct output |
| Time-sensitive news summaries | Avoid | Stale answers are worse than no answer |
| Multi-turn conversation | Medium | Cache only the first turn, not the full context |
When the Cache Lies: Failure Modes and Guardrails
The most dangerous failure mode is a false positive: two prompts that look similar but require different answers. A user asking "how do I deploy to production" and "how do I deploy to production with zero downtime" share most of their trigrams, but the second prompt needs a different response. The 0.92 threshold catches many of these cases, but not all, which is why the TTL and the drift audit exist as guardrails.
The second failure mode is context poisoning from cached responses that contain stale information. A response generated before a product change may remain in the cache for the full TTL, and every hit during that window propagates the outdated answer. Shortening the TTL reduces the risk but also reduces the hit rate, so the right value depends on how fast the underlying domain changes.
The third failure mode is cache poisoning from a bad response. If the endpoint returns an error message or a hallucinated answer, the cache stores it and serves it to every similar prompt until the TTL expires. The mitigation is to validate responses before caching, which can be as simple as checking that the response is non-empty and contains no error markers, or as complex as a separate quality classifier for high-stakes domains.
The Bottom Line
A semantic cache is the highest-leverage optimization available for free-tier LLM usage because it reduces both token consumption and latency without changing the model. The implementation shown here runs on any free server, requires no external dependencies, and pays for itself within the first hour of traffic. The threshold tuning and the drift audit are the price of safety, and they are cheap compared to the cost of duplicate requests.
Teams that are prototyping agents, building internal tools, or running evaluation loops on a budget should treat a semantic cache as a standard component rather than an optional enhancement. The same design works against any OpenAI-compatible endpoint, including MonkeyCode's free model access and free server option, which makes the comparison cheap to run. A cached token is a token that never needs to be spent, and on a free tier that distinction is the difference between a project that runs out of runway and one that keeps going.
