Building My First RAG System: Deriving the Architecture from First Principles - Part One

Intro

I recently read an article about a VC who uses AI to boost his productivity. He described building a knowledge base using NotebookLM, and one point that stuck with me was:

Every time I read something online that I thought I wanted to remember, I'd copy and paste it into that repository. Whenever I wanted to write a blog post, I could query it and retrieve all the information I needed.

Like him, I have knowledge and resources scattered across Logseq, Gmail, Notion, ADR documents, Slack, project readmes, Markdown files, Twitter, and more. That made me wonder: how could I build my own system? Tools like NotebookLM exist, but I want a single knowledge layer across all my sources—not isolated, manually managed workspaces. NotebookLM’s model requires creating a workspace, adding sources, and asking questions about them, but separate notebooks mean separate contexts.

As an experienced engineer who’s never built a Retrieval-Augmented Generation (RAG) system, I saw this as an opportunity to learn and share. I’ll approach it from first principles, and in this series, we’ll:

• Architect a RAG system from the ground up.
• Break its subsystems down and clarify their responsibilities.
• Identify architectural decisions and tradeoffs.
• Integrate the RAG system with an LLM to create something like a personal Google Search for your whole digital life.

Use Case

Two years ago, I read an article about a man with ADHD. The post stayed with me, but for over a year I couldn’t find it again, even after searching bookmarks and Googling "article about a guy with ADHD". I finally found it because the author emailed it to his mailing list. Without that email, I might never have seen it again.

With a personal knowledge base (RAG system), I could have simply asked for "an article about a guy with ADHD" and quickly found it. Let’s dive into how such a system works.

What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation is the process of supplementing LLM (Large Language Model) prompts with relevant information retrieved from your data. Instead of just relying on general training data, an LLM can answer your questions with reference to your actual information. The typical steps:

1. Relevant context is retrieved from a datastore of your sources.
2. The system augments your query with this context.
3. The LLM receives the augmented prompt and generates a response based on it.

Fundamentals

The core problem:

Find the right knowledge at any given time from multiple, diverse sources and provide good enough context for AI to answer questions.

To solve this from first principles, what capabilities are required? At a minimum, the system must:

• Access knowledge from different sources.
• Ingest knowledge into the system.
• Convert it to a consistent internal structure.
• Store and index it for efficient searching.
• Retrieve the most relevant information for any given question.
• Prepare that data as context for the LLM.
• Generate a useful response.

These lead straight to the major system components:

• Sources: Where information is stored in raw form (note apps, Notion, Markdown, LinkedIn, Twitter, ADR docs, etc.).
• Source Connectors: Components that communicate with the various sources and fetch data into the system.
• Ingestion: Talks to source connectors and brings raw data into the system.
• Processing: Normalizes diverse data formats into a uniform structure, similar to how APIs standardize on JSON.
• Knowledge Storage: Stores the processed data in structures optimized for fast retrieval. Depending on our retrieval needs, this may involve document storage, full-text indexes, vector indexes, or a combination of them.

• Retrieval: Finds the knowledge base items most relevant to the user's question. This is the heart of RAG. It ranks results by relevance. E.g., if you ask about an ADHD blog post:
  

    Blog post whose title/content strongly matches the ADHD query - very relevant
    Personal note discussing ADHD - relevant
    A tweet about ADHD - probably relevant
    Random tweet - probably irrelevant
  

For now, ranking can be bundled into retrieval; in larger systems, it might be a separate stage.

• Context Preparation: Builds the final prompt with user question and retrieved context for the LLM. This is where the “augmentation” happens.
• Generation: The LLM generates the answer, referencing the appropriate sources. This is the "G" in RAG.

flowchart TD
    A[Knowledge Sources]
    B[Source Connectors]
    C[Ingestion]
    D[Processing]
    E[Knowledge Storage / Indexing]
    F[Retrieval]
    G[Context Preparation]
    H[LLM Generation]
    I[Answer with References]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I

Conclusion

We’ve broken down the essential building blocks of a RAG system—without writing code yet. The aim was to clarify the problem so that the technical architecture emerges naturally.

If I want an AI system that can answer questions based on my dispersed personal knowledge, it must fetch that knowledge, ingest and process it into a standard form, store it accessibly, retrieve the most relevant parts, prepare them as LLM context, and use all this to generate useful responses.

Before now, I understood concepts like embeddings, vector databases, and indexing as isolated pieces. Taking a first-principles approach reframed it as a system design problem: how do I move knowledge from raw sources to usable LLM context?

Now, things like embeddings and indexing become tools for certain subproblems—not the whole system.

In the next part, we’ll look deeper at how these components work together, what the actual data flow looks like, and where important design decisions come in.