Large websites contain an enormous amount of information.
The problem isn't always finding information.
The problem is asking the right question against that information efficiently.
If you crawl a website and simply store the raw HTML, you haven't really built a useful knowledge system.
You still need to:
- Clean the content
- Remove unnecessary information
- Break the content into meaningful chunks
- Generate embeddings
- Store those embeddings
- Retrieve relevant information
- Generate an answer from the retrieved context
- Show users where that information came from
That was the motivation behind OmniScrape, a modular RAG engine I built using FastAPI and Qdrant.
The Problem
Imagine a website containing hundreds or thousands of pages.
A user might ask:
"What does this documentation say about authentication?"
Traditional keyword search can work well for simple queries.
But what happens when the user doesn't use the exact terminology from the website?
For example, the documentation might say:
"Requests must include a valid bearer token."
while the user asks:
"How do I authenticate my API requests?"
The words are different, but the meaning is related.
This is where Retrieval-Augmented Generation (RAG) becomes useful.
Instead of asking an LLM to answer entirely from its general knowledge, the system first retrieves relevant information from the target knowledge base and then gives that information to the model.
The basic idea is:
Question
↓
Retrieve relevant content
↓
Give retrieved content to LLM
↓
Generate answer
OmniScrape was designed around this architecture.
The Goal
The goal was to build a system that could take website content and turn it into a searchable knowledge base.
The pipeline needed to:
- Crawl websites
- Extract useful content
- Clean the content
- Split it into chunks
- Generate embeddings
- Store embeddings and metadata
- Retrieve relevant information
- Generate an answer
- Return citations to the original sources
The overall architecture looked roughly like this:
┌──────────────┐
│ Website │
└──────┬───────┘
↓
┌──────────────┐
│ Crawler │
└──────┬───────┘
↓
┌──────────────┐
│Content Clean │
└──────┬───────┘
↓
┌──────────────┐
│ Chunking │
└──────┬───────┘
↓
┌──────────────┐
│ Embeddings │
└──────┬───────┘
↓
┌──────────────┐
│ Qdrant │
└──────┬───────┘
↓
Retrieval
↓
┌──────────────┐
│ Answer Model │
└──────┬───────┘
↓
Answer + Sources
The important part is that this isn't one large AI operation.
Each stage has a specific responsibility.
Why Qdrant?
A vector database is one of the central components of a RAG system.
I chose Qdrant because it provides a dedicated vector search layer and allows embeddings to be stored together with useful metadata.
That metadata becomes particularly important when building source-aware retrieval.
A chunk shouldn't just be:
Some paragraph from a website...
It should also carry information about where that paragraph came from.
Conceptually, a stored record could look like:
{
"text": "Authentication can be configured...",
"source": "https://example.com/docs/authentication",
"title": "Authentication",
"chunk_id": "auth-004"
}
Now retrieval can return both the content and its provenance.
That becomes important later when generating citations.
Why FastAPI?
FastAPI was a natural fit for the backend because I wanted a clean API layer between the application and the RAG pipeline.
Instead of coupling the frontend directly to crawling, embedding, retrieval, and generation logic, the backend exposes clear endpoints.
Conceptually:
POST /crawl
POST /ingest
POST /query
GET /sources
The frontend doesn't need to know how the RAG engine works internally.
It only needs to communicate with the API.
This separation also makes the backend easier to test and extend.
The Ingestion Pipeline
Before a user can ask questions, the website needs to be converted into useful information.
This is where most of the RAG pipeline actually happens.
1. Crawl
The first step is retrieving pages from the target website.
But raw HTML isn't suitable for embedding.
A webpage can contain:
- Navigation menus
- Scripts
- Footers
- Advertisements
- Sidebars
- Repeated content
- Other elements unrelated to the actual information
So crawling is only the beginning.
2. Clean
The extracted content needs to be converted into meaningful text.
The objective is to preserve useful information while removing unnecessary page noise.
For example, instead of storing a page like:
Home
Products
Pricing
Documentation
Login
Authentication
Authentication allows applications...
Privacy
Terms
Copyright
the system should focus primarily on the useful content:
Authentication
Authentication allows applications...
This matters because irrelevant content can eventually affect retrieval quality.
3. Chunk
Long documents cannot simply be treated as one giant piece of information.
The content needs to be divided into smaller sections.
For example:
Documentation
├── Introduction
├── Authentication
├── Configuration
├── API Reference
└── Troubleshooting
Chunking looks simple, but it has a major effect on retrieval.
If chunks are too large, retrieval can return a lot of irrelevant information.
If chunks are too small, important context can be separated across multiple chunks.
So chunking becomes a tradeoff rather than a fixed rule.
4. Embed
Each chunk is converted into a vector representation.
The embedding captures semantic information about the content.
This allows the system to search based on meaning rather than requiring an exact keyword match.
For example:
User query:
"How do I authenticate API requests?"
↓
Semantic representation
↓
Vector search
↓
Relevant documentation
The user's wording doesn't need to exactly match the wording in the original documentation.
5. Store
The vectors and associated metadata are stored in Qdrant.
At this point, the website has effectively been transformed into a searchable semantic knowledge base.
Instead of searching thousands of raw pages every time a user asks a question, the system can search the vector representation of the processed content.
Retrieval
When a user asks a question, the query goes through a similar embedding process.
The system searches Qdrant for chunks that are semantically close to the query.
Conceptually:
User:
"How do I authenticate API requests?"
↓
Query embedding
↓
Qdrant similarity search
↓
Relevant documentation chunks
↓
LLM
The retrieved chunks become the context for answer generation.
This is the core of RAG.
But it also introduces one of the most important problems in RAG systems:
What happens if the system retrieves the wrong information?
Why Retrieval Quality Matters
A RAG system can generate a fluent answer and still be wrong.
The problem may not be the LLM.
The retrieval layer may simply have returned poor context.
For example:
Question
↓
Retriever
↓
┌──────────────────────────┐
│ Unrelated chunk │
│ Navigation content │
│ Old documentation │
│ Relevant chunk │
└──────────────────────────┘
↓
LLM
The model now has to generate an answer from a noisy context.
This is one of the most important lessons I learned while building OmniScrape:
Better generation cannot compensate for consistently bad retrieval.
This is why ingestion, chunking, metadata, and retrieval quality deserve as much attention as the final LLM.
Why Citations Matter
One of the features I considered particularly important was source awareness.
A generated answer without a source can be difficult to trust.
For example:
"API authentication requires a bearer token."
That might sound reasonable.
But where did it come from?
A better system can point the user back to the relevant source.
Instead of:
AI says something.
the experience becomes:
AI says something.
Source:
Authentication Documentation
The user can verify the information.
This is particularly useful for:
- Technical documentation
- Internal knowledge bases
- Research
- Support systems
- Company documentation
The goal isn't just to generate an answer.
The goal is to make the answer traceable.
Engineering Challenges
Building the pipeline also exposed several engineering challenges.
Chunk Size
There is no universal perfect chunk size.
Documentation, blog posts, technical specifications, and long-form articles can have very different structures.
The chunking strategy therefore needs to match the type of information being processed.
Metadata
Metadata initially looks like an implementation detail.
It isn't.
Without useful metadata, it becomes much harder to understand where retrieved content came from.
Useful metadata can include:
- Source URL
- Page title
- Section
- Chunk ID
- Other document-level information
This information becomes valuable when debugging retrieval and generating citations.
Retrieval Quality
Retrieval is one of the most important parts of the entire system.
If the wrong chunks are returned, the final answer can be wrong even if the LLM is working exactly as expected.
This means that evaluating a RAG system shouldn't stop at:
"Does the final answer sound good?"
A better question is:
"Did the system retrieve the right information to answer the question?"
Response Formatting
Even when the correct chunks are retrieved, the final response needs to be presented clearly.
The system needs to balance:
- Answer quality
- Source visibility
- Context length
- Response speed
- Readability
Too little context can produce weak answers.
Too much context can introduce noise.
Too many citations can make the response difficult to read.
The final response therefore needs its own design considerations.
Why Modular Design?
I intentionally separated the system into components.
Conceptually:
crawler/
├── crawler.py
ingestion/
├── cleaner.py
├── chunker.py
embeddings/
├── embedding_service.py
retrieval/
├── retriever.py
generation/
├── generator.py
api/
├── routes.py
frontend/
The exact implementation can change, but the principle stays the same:
Each component should have a clear responsibility.
This makes the system easier to evolve.
For example:
Changing the embedding model shouldn't require rewriting the crawler.
Changing the frontend shouldn't require changing vector storage.
Improving retrieval shouldn't require rebuilding the entire API.
This is one of the biggest advantages of modular AI engineering.
What OmniScrape Can Become
The architecture isn't limited to one specific website.
A system like OmniScrape can provide the foundation for:
- Internal knowledge search
- Documentation assistants
- Customer support tools
- Research assistants
- Technical knowledge bases
- Website question-answering systems
The core idea remains the same:
Unstructured Information
↓
Structured Knowledge
↓
Semantic Retrieval
↓
Source-Aware Answer
That makes the architecture reusable across different applications.
What I Learned From Building It
The biggest lesson from OmniScrape was that RAG isn't simply:
Vector Database + LLM
There is considerably more engineering involved.
Crawling quality affects ingestion.
Ingestion affects chunking.
Chunking affects embeddings.
Embeddings affect retrieval.
Retrieval affects the context given to the model.
And the context affects the final answer.
In other words:
A RAG system is a chain. If one part of the chain is weak, the final answer suffers.
That is what made this project particularly valuable to me.
I wasn't just building a chatbot.
I was learning how to build an information pipeline around an AI model.
Final Thoughts
Building OmniScrape changed the way I think about RAG applications.
The LLM is only one component.
The real system includes:
Crawling
↓
Content Processing
↓
Chunking
↓
Embeddings
↓
Vector Storage
↓
Retrieval
↓
Generation
↓
Sources
Every stage contributes to the final result.
And when something goes wrong, the solution isn't always to change the model.
Sometimes the problem is the data.
Sometimes it's the chunking.
Sometimes it's retrieval.
Sometimes it's the way the final context is constructed.
That's the part of RAG engineering I found most interesting:
Building a system where the model doesn't have to know everything—it just needs to receive the right information at the right time.
Tags
#ai #rag #python #fastapi #qdrant #webdev
