I Counted the Attack Vectors in Our AI Stack and Now I Can't Sleep

so a while back i did something i do not recommend before bed.

i counted the vendors in a normal production AI stack. not a weird stack. not a bad team's stack. the stack everyone builds, the one in every architecture diagram on this site.

model gateway. vector database. memory service. rag framework. embeddings provider. orchestration. observability. two or three model providers.

that's eight. for one app.

and here's the first thing that got me: nobody approved that stack. not one person at any company, ever, sat down and said "yes, eight vendors, sounds right." it got approved one sprint at a time. a ticket here, a free tier there, a "we'll swap it later" that became load-bearing.

your architecture wasn't designed. it accreted. like a reef. or plaque.

the fun part: what each one actually is

security people have a phrase, "attack surface," and it sounds abstract until you translate each vendor into what it really is:

a model gateway is a standing API key in your CI. a vector database is a second copy of your data. a memory vendor is your prompts in someone else's logs. a rag framework is somebody's SDK running inside your process. observability is prompt logs leaving your network as the product working correctly.

read that last one again. the exfiltration is the feature. you pay for it monthly.

quick quiz you will fail

open two lists. one: every AI line item in your billing. two: every AI-related package in your lockfiles.

do they match?

they never match. i've asked a lot of teams. the answer is always no, and everyone laughs, and it's the nervous kind of laugh, because the gap between those two lists is software that touches your customer data and appears in exactly zero of your security reviews.

the gap is the attack surface. congratulations, you just did a security audit with grep.

"embeddings are just numbers though"

this is my favorite myth in all of AI, because it's so comforting and so wrong.

the theory: we embedded the text, so now it's a vector, vectors are anonymous math, we're fine.

the practice: embedding inversion is published research. morris et al. (2023, "text embeddings reveal (almost) as much as text") reconstructed the exact input text from its embedding 92% of the time for short inputs. not "similar vibes." the actual text.

so your vector database is not a bag of anonymous numbers. it's a second, queryable database of your source data, with its own API key, sitting in someone else's cloud.

now the aha that hurts: go ask whoever maintains your data inventory whether the vector store is on it.

it's not on it. it's never on it. you have a database of customer data that your own compliance team doesn't know exists, and you built it on purpose, with a jira ticket.

deletion theater

follow one user message through the stack. it hits the gateway (logged). gets embedded (copy two, the vector db). the memory service persists it (copy three). observability captures the full trace (copy four).

one message. four systems. four retention policies you've never read.

now legal walks over: "customer invoked their right to erasure, delete their data."

you cannot do that. what you can do is file four vendor tickets and hope. under gdpr this is the difference between "we called the delete endpoint and logged the result" and "we asked four companies nicely." one of those survives an audit. guess which one you have.

i call this deletion theater. everyone has a retention policy. almost nobody has a delete button that actually reaches all four copies.

deletion you can't prove is deletion you don't have. put it on a poster.

meanwhile, your dependencies

your rag framework is not one dependency. it's a tree. hundreds of packages, written by strangers, pinned loosely, executing in the same process that holds your database credentials.

this isn't hypothetical doom. langchain shipped a remote code execution bug in its math chain (CVE-2023-29374). pytorch's nightly build got hit by dependency confusion in 2022. owasp made supply chain a whole category in its LLM top 10 because this keeps happening.

the framework's security posture is your security posture. you married it. you just didn't have a ceremony.

and the stack you didn't even choose

everything above is the stack you deployed on purpose. there's another one.

it's your employees pasting contracts, code, and customer records into whatever consumer chatbot they like, on their personal accounts, right now, while you read this.

your company probably "banned" this. here's the aha: the ban is working as a compliance document and failing as a control. people don't stop using tools that make them faster. they stop telling you.

the only fix i've seen work is replacement, not prohibition. one sanctioned surface, the models people actually want, behind sso, logged. you can't firewall your way out of shadow AI. you can only out-compete it.

so what do you actually do

the counterintuitive answer, and the whole point: the fix is not more security. it's less stack.

every security tool you bolt onto an eight-vendor sprawl is a ninth vendor. the arithmetic only moves in one direction: fewer systems holding data, fewer standing keys, fewer egress paths. concretely, in priority order:

fewer copies. every system holding prompts or embeddings is a copy. count them, then make the number smaller.
one egress door. all model traffic through one brokered path where policy lives. no SDK in app code dialing providers directly.
one control plane, your keys. one place to rotate, one audit trail that answers "which model saw which data." byok, so consolidating the plane never means handing over the keys.
deletion as an api call. if you can't prove deletion programmatically across every copy, see poster above. you can build all of this yourself with enough glue code and discipline. some teams genuinely do. it's a lot of glue.

the horse, as promised

this counting exercise is roughly why backboard exists. routing across 17,000+ models, memory (first on the LoCoMo and LongMemEval benchmarks, receipts on github), embeddings, rag, and stateful threads behind one api and one key. for enterprises it ships as a signed container inside your own cloud, behind the IAM and SOC and audit trail you already trust. egress to outside models is opt-in per workload, never standing. deletion is an endpoint, not a ticket.

the objection i get, and it's a fair one: "congrats, you built a single point of compromise." three answers. byok means the platform never holds your keys, so consolidating the control plane isn't consolidating the credentials. in the enterprise deployment the one door is your door, inside your perimeter. and memory is exportable and routing is model-agnostic, so the exit is always open, which is the only honest answer to lock-in.

also, you'll notice zero certification badges in this post. on purpose. we publish nothing we haven't earned. ask for the architecture, not the badge.