I wanted a RAG system that was fast to run and fast to set up for clients. Upload a PDF, ask questions, get answers with citations. Pretty standard stuff for anyone freelancing in the AI space.

The problem is that every tutorial I found stops at a Jupyter notebook. Working demo, zero production readiness. No auth, no caching, no way to handle more than one user. The happy path, and nothing else.

So I built the whole thing. Deployed, running, something I can actually show to clients.

Here's what that looked like.

Live Demo | Backend Repo | Frontend Repo

Following a question through the pipeline

The easiest way to explain this system is to trace what happens when someone asks a question.

Say a user uploaded a 150-page book and types: "What's the main argument of chapter 3?"

The fingerprint problem

First thing I need to figure out: have I seen this question before?

Not this exact string. That's easy. But what about "what is the main argument of chapter 3" without the contraction? Or "What's the main argument of chapter 3??" with extra punctuation? Different casing?

Same question, four different strings. A naive string comparison treats each one as unique, and each one costs an OpenAI embedding call. That adds up.

The fix I landed on: normalize the query first. Expand contractions, strip punctuation, remove stopwords, collapse whitespace. Then SHA-256 the cleaned string.

The normalization + fingerprinting code

function normalizeQuery(input: string): string {
    let text = input.toLowerCase().trim();

    // "don't" → "do not", "what's" → "what is"
    for (const [k, v] of Object.entries(CONTRACTIONS)) {
        text = text.replace(new RegExp(`\\b${k}\\b`, 'g'), v);
    }

    text = text.replace(/[^a-z0-9\s]/g, ' '); // strip punctuation
    text = text.replace(/\s+/g, ' ');           // collapse spaces

    const words = text.split(' ').filter(w => !STOPWORDS.has(w));
    return words.join(' ').trim();
}

const fingerprint = crypto.createHash('sha256')
    .update(normalizeQuery(query))
    .digest('hex');

All four variations produce the same hash. One fingerprint, one cache key. The embedding cache and response cache both use this, so every performance shortcut downstream starts here.

Cache check

The pipeline looks in Redis for a cached embedding vector at emb:{fingerprint}. If it finds a 1536-dimensional array there, it skips the OpenAI embedding call. About 200ms saved before we even talk to the vector database.

Embedding cache logic

let queryEmbedding: number[];
const cached = await cacheService.getCache(`emb:${fingerprint}`);

if (cached) {
    queryEmbedding = JSON.parse(cached);
} else {
    queryEmbedding = await embeddingService.getEmbedding(query);
    await cacheService.setCache(
        `emb:${fingerprint}`,
        JSON.stringify(queryEmbedding),
    );
}

There's also a response cache (resp:{fingerprint}) that stores complete LLM answers. I found out during testing that this breaks in conversation mode, though. The same question asked as a follow-up has different context because of chat history, so the cached answer would be wrong. The response cache only kicks in for standalone queries. The embedding cache works everywhere.

I added an X-Cache-Embed response header so the frontend knows which path was taken. Helps when debugging latency issues in production.

Vector search

The embedding goes to Pinecone for similarity search. Multi-tenancy is where things get interesting.

Most tutorials scope queries with metadata filters: where: { userId: 'abc' }. That works until someone forgets the filter and suddenly users can see each other's documents.

I used Pinecone namespaces instead. Each user's vectors live in a separate namespace. It's not a filter you can forget to add. The data is physically separated.

Namespace-scoped operations

// Every operation is scoped to the user's namespace
await this.index()
    .namespace(userId)
    .upsert({ records: vectors.slice(i, i + BATCH) });

const result = await this.index()
    .namespace(userId)
    .query({ vector: queryEmbedding, topK, includeMetadata: true });

await this.index()
    .namespace(userId)
    .deleteMany({ ids });

Redacting context before it reaches the LLM

Pinecone returns the top-K matching chunks. Before they go to the LLM, there's a step most tutorials ignore completely.

If someone uploads a security doc with phrases like "bypass authentication" or "disable firewall rules," and asks the right question, the LLM will repeat those instructions. It's in the prompt context. The model doesn't know it shouldn't say that.

So I scan the retrieved chunks for suspicious patterns and replace them with [REDACTED_REASON] before the LLM ever sees the context. The response includes a policy field (allow or partial) so the client knows whether redaction happened.

Pre-LLM redaction filter

const preFilterDocs = (text: string) => {
    const suspicious =
        /\b(bypass|disable|ignore rules|unrestricted|open firewall|run arbitrary)\b/gi;
    const redacted = text.replace(suspicious, '[REDACTED_REASON]');
    const found = suspicious.test(text);
    return { redacted, found };
};

const policyCheck = () => {
    const hasSuspicious = preFilterResults.some(r => r.prefilter.found);
    if (hasSuspicious)
        return { decision: 'partial', reason: 'context_redacted' };
    return { decision: 'allow', reason: 'ok' };
};

Streaming the answer

The redacted context hits GPT-4o-mini. Total response time is about 5.9 seconds, which sounds slow, but the user sees text after 3.5 seconds because of streaming. They're already reading before the model finishes.

The tricky part: source citations need to come after the stream ends. I split it into two SSE event types.

The streaming protocol

// Stream LLM chunks as they arrive
await llmService.streamAnswer(query, redactedContext, (chunk) => {
    fullAnswer += chunk;
    res.write(`data: ${JSON.stringify({ type: 'chunk', data: chunk })}\n\n`);
});

// Final event: source citations + policy decision
res.write(`data: ${JSON.stringify({
    type: 'done',
    provenance,
    policy: { decision: policyResult.decision, reason: policyResult.reason },
})}\n\n`);
res.end();

chunk events build the message in real time. done triggers the source citation chips below the answer. Two event types, one connection.

Frontend problems I didn't expect

The backend was the interesting part. The frontend is where things broke in ways I didn't anticipate.

Streaming and navigation don't mix in Next.js. When a user sends their first message, the app creates a conversation and updates the URL from /chat to /chat/abc123. I used router.replace(). It unmounted the component. The SSE connection died. The user saw half an answer and got bounced to an error page.

Fix: window.history.replaceState(). Updates the URL without triggering React navigation. Component stays mounted, stream keeps going.

There's also this: the backend always returns source citations even when it says "I don't know." It found context but couldn't answer. Showing citations under a non-answer confuses people. The frontend checks the response text against a regex and hides the source chips when the answer looks like a "no knowledge" response. Took me longer than it should have.

Going to production

Getting this running locally was one thing. Deploying it was a different exercise.

The first Dockerfile was 900MB and ran the dev server. After a few rounds, I landed on a multi-stage build: TypeScript compiles in one stage, the production image only gets the compiled JavaScript and prod dependencies. About 250MB.

First Railway deployment crashed because uploads/ didn't exist in the container. I'd put it in .dockerignore (you don't want local PDFs in your image), but multer needs that directory to write temp files. One-line fix: fs.mkdirSync('uploads', { recursive: true }) at startup. Then I realized the files should be deleted after indexing anyway. No persistent storage, no S3, no disk space to worry about.

The health endpoint hits Postgres, Redis, and Pinecone in parallel with individual timeouts. Railway restarts the container if it fails. Graceful shutdown drains connections with a 10-second limit. CORS is wide open locally, locked to the Vercel domain in production.

None of this is interesting, but skip any of it and the deploy falls over.

Numbers

From the production deployment:

Most of the latency is OpenAI. Embedding generation and LLM inference, not server compute. Streaming covers the gap: 3.5 seconds to first visible text, and people start reading.

Tech stack

What I'd change

My chunks are fixed at 500 tokens with 100 overlap. They cut mid-sentence sometimes, and the LLM struggles with the partial context. I'd switch to semantic chunking, splitting on paragraph boundaries instead.

I wrote a reranking service and then never connected it. A cross-encoder between vector search and the LLM would drop the marginal chunks before they eat up context window. It's sitting there, just not plugged in.

The other thing: if the LLM errors mid-stream, the user sees a half-finished message and nothing else. No error state, no retry button. I need a type: 'error' event in the SSE protocol. It's on my list.

TypeScript, Express, Next.js, OpenAI, Pinecone, Redis, PostgreSQL, Clerk. Railway + Vercel. About $5-8/month.

If you have questions about any of the architecture decisions, I'm happy to talk about them in the comments.