The leak you end up paying for is rarely the request body; it is the log line the request left behind. Outbound traffic gets a threat model, a reviewer, and a deny list, while the logger gets a one-liner and a retention setting nobody remembers choosing. The workflow below keeps the debugging power and removes the second copy: every model call writes a shape record instead of content, and CI refuses any log file that disagrees. If you change one thing about your AI feature this quarter, change what your logs are allowed to contain.
A shape record answers the questions you actually ask
During an incident you ask four things: did the prompt grow, which fields appeared, was the call retried, and how slow was the model. None of those questions need the text. The reflex is to encrypt logs, but encryption protects the disk and not the pipeline, because search indexes, support bundles, and dashboard exports all decrypt on read. A shape record survives that pipeline unchanged, since there is genuinely nothing inside it to decrypt.
# shape_log.py — emit debug-grade metadata, never message content
import hashlib, json, time
SECRET_KEYS = {"authorization", "cookie", "x-api-key", "set-cookie"}
def shape(record: dict, model: str, latency_ms: int, attempt: int = 1) -> dict:
payload = json.dumps(record, sort_keys=True, ensure_ascii=False).encode("utf-8")
return {
"event": "model_call",
"model": model,
"attempt": attempt,
"payload_digest": hashlib.sha256(payload).hexdigest()[:16],
"payload_bytes": len(payload),
"field_names": sorted(k for k in record if k.lower() not in SECRET_KEYS),
"latency_ms": latency_ms,
"ts": round(time.time(), 3),
}
# emit as JSONL, one call per line
with open("logs/model.jsonl", "a", encoding="utf-8") as fh:
fh.write(json.dumps(shape({"prompt": "...", "user_email": "..."}, "default", 812)) + "\n")
Two fields do most of the work, and both are cheap. The digest gives you a stable identifier that joins retries and duplicates without telling you what was sent, so a cache hit and a silent re-prompt look different in the log. The field-name list is the cheapest schema test you will ever write: a stray user_email key shows up as a name rather than an address, which tells you a serializer changed without showing you anyone's data.
The gate is a test, not a policy document
A written rule that logs must not contain prompts will be violated within a sprint, because nobody runs the rule before merging. Turn it into a test that reads your fixture directory and fails the build, then keep the fixture set small enough that reviewers actually open it.
# tests/test_logs_are_shape_only.py
import json, pathlib, re
PATTERNS = {
"email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"),
"bearer": re.compile(r"\bBearer\s+[A-Za-z0-9._-]{12,}"),
"aws_key": re.compile(r"\bAKIA[0-9A-Z]{16}\b"),
"private_key": re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
}
ALLOWED = {"event", "model", "attempt", "payload_digest", "payload_bytes",
"field_names", "latency_ms", "ts"}
def test_shape_only():
for path in pathlib.Path("tests/fixtures/logs").glob("*.jsonl"):
for line in path.read_text().splitlines():
if not line.strip():
continue
rec = json.loads(line)
extra = set(rec) - ALLOWED
assert not extra, f"{path.name}: unexpected field(s) {extra}"
blob = json.dumps(rec)
for name, pat in PATTERNS.items():
assert not pat.search(blob), f"{path.name}: {name} pattern leaked"
Keep one deliberately leaky fixture in that directory and assert the scanner flags it, otherwise you are trusting a regular expression nobody ever executed against a failure. Run the gate locally before you push, since the CI failure is more useful as a reminder than as a red build at 2am.
python -m pytest tests/test_logs_are_shape_only.py -q
# how many shape records did your last incident actually need?
jq -r 'select(.event=="model_call") | .payload_digest' logs/model.jsonl \
| sort | uniq -c | sort -rn | head -20
Deleting by digest is the part people skip
Shape records still age, and stale metadata is where the quiet risk lives. A digest plus a timestamp is linkable, and linkable metadata is personal data under several regimes, so a hash is not an anonymization strategy; it is a retention strategy. When an incident closes, take the digests you collected and sweep every derived artifact — cache keys, eval fixtures, support bundles, the notebook someone exported — with the same identifier.
def sweep(store, digests: set[str]) -> int:
dropped = 0
for key in list(store.keys()):
if store[key].get("payload_digest") in digests:
del store[key]
dropped += 1
return dropped
The reason to run this harness on a throwaway environment is frequency rather than price, because a gate you only run before a release teaches you very little. MonkeyCode's free model access and free server option let you keep the mirror environment and the model calls inside a sandbox you delete after each run, and that disposable property matters more than the cost, since fixtures shaped like real traffic should never live on a host you intend to keep.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Where this approach breaks
Digests are not pseudonyms for low-entropy inputs, and a digest of yes, a date, or an address drawn from a small customer list is brute-forceable in seconds. Field names also leak more than teams expect, because patient_dob in a prompt describes the data even when the value is absent, so the allow-list needs review whenever a serializer changes. Shape records cannot reproduce an exact output either, which means you need a separate, opt-in reproduction path with explicit consent instead of quietly widening the log.
Skip this workflow if a regulation requires content-level audit trails you cannot reconstruct from metadata, and skip it if your only debugging tool is a full local trace with no other reproduction route. Very small projects with a single developer and no user data are also fine with plain logs, provided the retention window is short and someone wrote it down. For everyone else, the shape record costs an afternoon and removes a category of incident report you never want to file.
If you want to test the idea, stand it up against a copy of your logs on a free tier, run the gate once, and delete the environment the same day.
