What I Got Wrong Building a RAG Pipeline from Scratch in TypeScript

I'm building a production-grade, multi-tenant AI support agent in TypeScript. No Python. No LangChain. Just Node.js, PostgreSQL with pgvector, and a lot of wrong assumptions that taught me more than any tutorial.

This post covers three things I got wrong while building my RAG (Retrieval-Augmented Generation) pipeline, what I replaced them with, and why the replacements work. If you're building RAG systems , especially in TypeScript , this might save you a few days of debugging.

The full codebase is open source: helpdesk-ai on GitHub

First, what is RAG?

LLMs don't know your data. They were trained on public internet text, not your company's docs. RAG is the pattern that fixes this:

1. Ingest your documents , split them into chunks, convert each chunk into a numeric vector (an "embedding") that captures its meaning
2. Store those vectors in a database that supports similarity search (pgvector, Pinecone, Weaviate, etc.)
3. Retrieve when a user asks a question, embed the question the same way, find the most similar chunks
4. Generate stuff those chunks into the LLM's context as reference material, and let it answer based on real data instead of guessing

Simple concept. The devil is in step 1 and step 3.

Mistake #1: Fixed-size chunking is deceptively bad

The most common RAG tutorial tells you to split documents into chunks of N characters with some overlap. It's three lines of code and it "works."

Here's what it actually does to a real document. Imagine a markdown doc with a heading "Setting up Stripe webhooks", followed by an explanation, followed by a TypeScript code block that initializes Stripe and starts a webhook handler. A naive fixed-size chunker will happily cut the document mid-code-block, leaving you with two chunks:

• Chunk 1: An explanation plus the first half of a code snippet that initializes Stripe but doesn't show the webhook handler
• Chunk 2: The second half of the code , req.body, sig, endpointSecret , with zero context about what it's doing

When the embedding model processes chunk 2, it sees a fragment. The embedding it produces is incoherent it captures "something about request bodies and secrets" but not "Stripe webhook verification." When a user asks "how do I verify Stripe webhooks?", this chunk might not even surface. And if it does, the LLM gets half a code block and either hallucinates the rest or gives a confused answer.

What I did instead: structural chunking

Documents written by humans have structure. Headings exist for a reason , each section under a heading is a coherent unit about one topic. My chunker respects that:

// Split at heading boundaries (##, ###, etc.)
// Everything between two headings = one chunk
// Track the heading hierarchy for metadata

The algorithm:

1. Parse YAML frontmatter, extract the document title
2. Find all headings (# through ######), but skip headings inside code blocks (triple-backtick fences)
3. Split at heading boundaries , content between heading A and heading B is one chunk
4. Track the heading hierarchy as a "section path": "Stripe webhooks > Choosing a sync mode"
5. If a section exceeds 4000 characters (embedding model limit), sub-split at paragraph boundaries first, then line boundaries as fallback

The result: every chunk is a coherent unit that a human wrote as a coherent unit. The embedding captures the actual topic, not a random slice of text.

{
  "id": "a8f3e2d1b9c04567",
  "tenant_id": "posthog",
  "source_file": "published/docs/cdp/sources/stripe.md",
  "doc_title": "Linking Stripe as a source",
  "section_path": "Linking Stripe as a source > Choosing a sync mode",
  "content": "The actual section content...",
  "doc_type": "docs"
}

Key metadata decisions:

• section_path the heading hierarchy. Not just for display: it gives the LLM context about where in the document this chunk came from. "Choosing a sync mode" alone is ambiguous. "Stripe > Choosing a sync mode" is not.
• tenant_id baked in from day one. Every query scopes to a tenant. Hardcoded "posthog" now, per-tenant later. Multi-tenancy is cheaper to build in than to bolt on.
• Deterministic IDs via sha256(sectionPath + content) re-running the pipeline upserts (updates existing rows) instead of duplicating every chunk. This is idempotency: run it twice, get the same result.

The chunk size trap I fell into

My structural chunker worked great , until I ran the embedding step and 3 chunks silently failed. The embedding model (nomic-embed-text) has an 8192-token context limit. Three of my sections were massive , one was 13,409 characters.

The tempting fix: truncate in the embedding step. Just chop at 8192 tokens and embed what fits.

The correct fix: fix the chunker. Truncation silently loses content. A 13k-char section about webhook configuration would lose its entire second half , the part about error handling and retries. Instead, I added a two-layer splitting strategy:

1. Paragraph split (split on \n\n) preserves semantic coherence. Each paragraph is usually about one thing.
2. Line split (split on \n) fallback for content that's one giant paragraph, like tables or long code blocks.

Both use the same accumulateChunks function: buffer content until the next piece would exceed the limit, flush the buffer as a chunk, continue. DRY between both layers.

The principle: the embed step should trust that chunks are well-sized. Separation of concerns , the chunker's job is sizing, the embedder's job is embedding.

Mistake #2: Pure vector search has a blind spot

Vector search (semantic search) works by comparing the meaning of your query to the meaning of each chunk. "How do I set up Stripe?" and "configuring Stripe integration" have similar embeddings even though they share almost no words. That's the magic.

Here's the blind spot: vector search is fuzzy. It finds semantically similar content, but it can miss exact keyword matches that a user clearly intended.

I discovered this when I searched for "Stripe webhook" and got back results about Paddle and Shopify , semantically related (they're all payment integrations) but not what was asked for. The chunk specifically about Stripe webhooks ranked lower because its embedding was similar to many payment-related chunks.

Meanwhile, a simple text search for the word "Stripe" would have nailed it instantly.

What I did instead: hybrid retrieval with Reciprocal Rank Fusion

I run two searches in parallel:

1. Vector search embed the query, find nearest neighbors by cosine similarity in pgvector
2. Keyword search PostgreSQL full-text search (tsvector + ts_rank) using BM25-style scoring

Then I merge the two ranked lists using Reciprocal Rank Fusion (RRF):

score = weight * (1 / (rank + 1 + k))

Where k=60 is a smoothing constant. Each chunk gets an RRF score from both lists, and the final ranking is the sum.

Why RRF instead of just averaging the scores?

Vector similarity scores range 0–1 (cosine). Full-text search scores are arbitrary floats from ts_rank. They're not comparable. Averaging 0.85 (cosine) with 14.3 (ts_rank) is meaningless.

RRF sidesteps this entirely by using rank position, not raw scores. "Ranked 1st in vector + ranked 3rd in text" is comparable to "ranked 2nd in vector + ranked 1st in text" , regardless of the underlying score scales.

Why k=60?

Without the constant (k=0), rank 1 gets score 1.0 and rank 2 gets 0.5, a massive cliff. With k=60, rank 1 gets 1/62 and rank 2 gets 1/63 , a gentle slope. This means agreement between the two retrievers matters more than rank position in either one. A chunk that both retrievers think is relevant (rank 5 + rank 5) beats a chunk that only one retriever likes (rank 1 + rank 50).

The implementation detail that matters

I pre-compute tsvector at ingestion time, not at query time:

INSERT INTO chunks (..., search_vector)
VALUES (..., to_tsvector('english', $title || ' ' || $content))

Computing to_tsvector for every row on every query would be O(n). Pre-computing means the work happens once per chunk, and a GIN index makes lookups O(log n).

I also include the doc_title in the search vector. Without it, a chunk about Stripe that doesn't mention "Stripe" in the body text would be invisible to keyword search. The title carries topic-level signal that the body might not repeat.

Fetching enough candidates: I fetch topK * 4 candidates from each retriever, not just topK. A chunk ranked 6th in vector but 1st in text has a high RRF score , but would be missed entirely if I only fetched the top 5 from each list.

Mistake #3: Small models can't do tool calling (and it's not a quality issue)

This one wasn't about RAG directly, but it shaped how I built the agent that uses RAG.

An agent is an LLM in a loop. It reasons about what to do, calls a tool (like search_knowledge), sees the result, and decides what to do next. The LLM doesn't execute tools, it outputs structured JSON saying "please call function X with arguments Y." Your code executes it and feeds the result back.

I started with Mistral 7B running locally via Ollama. Asked the agent: "How do I set up Stripe in PostHog?"

Expected: the model calls search_knowledge({ query: "Stripe setup PostHog" })

What actually happened: the model wrote a paragraph explaining what it would do if it could call tools. It described the tool it would use, the arguments it would pass, even the results it expected to get. But it never emitted the structured tool_calls JSON. It narrated instead of acting.

Switched to qwen2.5:7b: same problem, slightly better. Switched to Groq's llama-3.3-70b-versatile , worked perfectly on the first try.

Why this happens: tool calling requires the model to output a specific JSON structure instead of natural language. It's a learned behavior from fine-tuning, not an emergent capability. A 7B model simply hasn't seen enough examples of structured tool-call output during training. A 70B model has. No amount of prompt engineering ("you MUST use tools, NEVER describe what you'd do") fixes a model that can't reliably emit the format.

Why this matters for architecture:

Model selection is an engineering decision, not just a quality preference. A model that produces beautiful English but can't emit tool_calls JSON is useless for agents. When I evaluate models for this system, tool-calling reliability is the first filter, before cost, before speed, before answer quality. A fast, cheap model that can't call tools is a fast, cheap paperweight.

This also affected tool selection at runtime. Even with qwen2.5:7b, the model picked search_text (grep over local files) instead of search_knowledge (RAG retrieval) for documentation questions. I fixed this with explicit guidance in the system prompt:

Use search_knowledge FIRST for product features, setup, and troubleshooting.
Use search_text only for local code files on disk.
If no relevant results found, say "I don't have documentation for that."

That last line is critical. Without it, when the agent couldn't find relevant docs, it hallucinated an entire Python implementation of S3 caching. With the instruction, it admits the gap instead of fabricating an answer.

What I'd do differently

Start with retrieval evaluation. I built the entire pipeline , ingestion, chunking, embedding, retrieval, agent integration, before measuring whether my chunks were any good. I should have built a small golden dataset (20 queries with expected chunks) after chunking and tested retrieval quality before wiring up the agent. Finding out your chunks are bad after building 6 packages on top of them is expensive.

Use a real tokenizer from day one. I used chars / 4 as a proxy for token count. It works for English prose. It breaks for code blocks (where one token can be one character) and unicode. When I eventually hit the embedding model's token limit, it was because my character-based estimate was wrong. A proper tokenizer would have caught oversized chunks at chunking time.

Chunk overlap for tables and lists. When my splitter cuts a table across two chunks, chunk 2 has data rows but no column headers. The embedding model doesn't know what the numbers mean. Standard mitigation is chunk overlap , repeat the header row at the start of each sub-chunk. I deferred this and it's now in my TODO list.

The stack

For anyone building RAG in TypeScript:

• Runtime: Node.js with Bun
• Embedding model: nomic-embed-text via Ollama (local, free, 768 dimensions)
• Vector store: PostgreSQL 16 + pgvector (HNSW index for approximate nearest neighbor)
• Full-text search: PostgreSQL's built-in tsvector + GIN index
• Agent framework: Hand-rolled ReAct loop (migrating to Mastra next)
• LLM providers: Ollama (local), Groq, Gemini , all via one OpenAI-compatible provider

The entire pipeline runs locally with zero API costs during development. Swap the embedding model and LLM provider for production , the architecture doesn't change.

What's next

• Mastra integration comparing my hand-rolled agent loop against a framework. What does Mastra abstract? Where does the abstraction leak?
• Evaluation suite 60+ golden test cases, programmatic + LLM-as-judge grading, regression testing when I change prompts or chunking
• Multi-tenant isolation per-tenant prompts, per-tenant knowledge base, per-tenant cost tracking. The tenant_id field is already on every chunk, but the agent itself doesn't scope anything yet.

If you want to dig into the code, the repo is structured as a Turborepo monorepo with clean package boundaries: agent-core (the loop), retrieval (hybrid search), ingestion (chunking pipeline), shared (embedding), and types.

Check out the repo | Twitter | LinkedIn

I'm Ayush Nautiyal, a backend engineer learning AI agent engineering in public. I write about what I build, including the parts that break. If you're hiring for AI/agent engineering roles (TypeScript, remote), I'd love to chat.