I keep running into the same conversation with people building internal AI tools for organisations that have more than one department: HR, finance, engineering, legal, whatever the split is. Everyone wants one server. Buying a separate GPU box per department is expensive, and most departments don't come close to using a full GPU's throughput for most of the working day. So the instinct is reasonable: share the hardware, share the model, and keep each department's data apart with a tenant_id column and a WHERE clause.
That instinct is where most of the trouble starts. A tenant_id column is access control. It is not isolation. Those are different guarantees, and conflating them is how an HR salary review ends up summarised into a finance dashboard, or a legal team's privileged documents get pulled into an engineering support bot's answer.
Namespaces are the actual boundary
Access control answers "is this request allowed to see this row." Isolation answers "can this data physically end up somewhere it shouldn't, regardless of whether a check was written correctly." On a shared server you need both, and they live in different places.
In practice that means treating each department as its own namespace at every layer the data touches, not just the database table:
- Storage: separate buckets or volume prefixes per department, not one bucket with a folder convention everyone is trusted to respect.
- Vector indexes: a separate index or namespace per department for retrieval, not one shared index with a metadata filter bolted onto every query.
- Model context: no shared conversation or prompt cache keyed only by a session ID, especially if session IDs can collide or be guessed across tenants.
- Credentials: a separate service identity per department with its own scoped key, not one shared service account that fans out to several departments relying on row filters to decide what it's "supposed" to return.
- Logging and audit trails: per-namespace logs, so a department's own compliance team can prove what left their boundary without trusting your application code to have tagged everything correctly after the fact.
None of this is exotic. It's the same multi-tenancy discipline you'd apply to any SaaS product. The difference on an internal, on-premise deployment is that people often skip it, because "it's all one organisation" feels like it lowers the stakes. It doesn't. Internal departments routinely have stricter mutual confidentiality requirements, HR and legal data about an ongoing dispute, finance data ahead of results, than a SaaS vendor typically owes its external customers.
Why row isolation alone breaks: a concrete example
Here's a failure mode I've seen play out almost exactly like this. A shared vector database holds embeddings for every department's documents, filtered at query time by tenant_id in application code. It works fine for the main chat interface, because every query path there remembers to filter.
Then someone builds a second feature: a nightly job that re-embeds documents after a model upgrade, or a debugging tool that lets an admin "search everything" to diagnose a bad retrieval result, or a caching layer that stores the top results for a popular query to save latency. None of those is the main chat interface. Each one is a reasonable thing to build on its own. And each one is a separate, independent place where someone has to remember to apply the same filter, correctly, forever, as the codebase changes under them.
Eventually one of them doesn't. The debugging tool surfaces a chunk of finance data on an HR analyst's screen while they're chasing an unrelated bug. The cache key doesn't include tenant_id, so two departments asking a similarly worded question get served each other's cached result, source chunks included. Nothing throws an error. No log line flags it. The data just quietly crosses a boundary that only ever existed as a convention in application code, never as a structural property of the system. That's the core weakness of database-level isolation on its own: its strength equals the weakest query path that ever touches the data, including ones that don't exist yet.
What a structural fix looks like
The shape of the fix is to move the isolation boundary out of query logic and into the object being queried, so no code path, present or future, can bypass it by accident:
# Fragile: isolation depends on every caller remembering a filter
def search_documents(tenant_id, query_vector):
results = vector_db.similarity_search(query_vector, top_k=10)
return [r for r in results if r.metadata.get("tenant_id") == tenant_id]
# a new caller (cache warmer, admin tool, batch job) that forgets this
# line compiles fine, runs fine, and silently leaks data across tenants
# Structural: isolation is a property of the object, not the query
def get_tenant_index(tenant_id):
return vector_db.get_namespace(tenant_id) # distinct index, distinct keys
def search_documents(tenant_id, query_vector):
index = get_tenant_index(tenant_id)
return index.similarity_search(query_vector, top_k=10)
# there's no "forgot the filter" failure mode here, because the
# connection object has no way to see another tenant's namespace at all
Physical separation vs shared infrastructure: the real tradeoffs
Once you accept that correct isolation takes real engineering effort, full physical separation starts to look tempting again: give every department its own server, and the failure I just described becomes structurally impossible, because there's no shared database for a bad query to reach in the first place.
Full physical separation has real advantages. It's the easiest architecture to explain to a compliance team or an auditor, because the boundary is a locked server room door rather than a line of code. A bug or an outage in one department's stack can't touch another's. And if one department genuinely needs air-gapped handling, classified government data is the obvious example, physical separation stops being a tradeoff and becomes the requirement, full stop.
But full physical separation has a cost people underestimate until they're paying it monthly. A GPU sized for one department's peak load sits mostly idle the rest of the time, and you pay for that idle capacity once for every department that gets its own box. You're also multiplying operational load: a model deployment to patch per department, a monitoring and alerting setup per department, a place a model upgrade can quietly go wrong, times however many departments you have, instead of once.
Shared infrastructure with strict, structural isolation (separate namespaces, separate containers or process-level sandboxing, separate encryption keys per department, all enforced at the object level rather than the query level) gets you most of the security properties of physical separation while sharing the expensive, often-idle GPU across departments whose peak usage rarely lines up at the same moment. The honest cost is that it's harder to build correctly and harder to verify from the outside. A shared host OS or hypervisor compromise is a shared blast radius in a way that physically separate boxes never are, no matter how good your namespace discipline is above that layer.
The working rule I've settled on when building this kind of thing: structural isolation, not row filters, by default on shared hardware, with full physical separation reserved specifically for workloads that carry an actual regulatory or classification requirement for it, rather than deployed as a default reaction to "this data feels sensitive."
I'd be genuinely interested to hear where other people draw that line in practice. At what point does structurally isolated shared hardware stop being good enough for you, and what's actually pushed you to pay for full physical separation rather than just isolating harder on the box you've already got?
