Large Language Models are powerful.
But there is one fundamental limitation:
An LLM doesn't automatically know your application's private data.
Your company policies, product documentation, internal knowledge base, customer records, PDFs, technical documentation, or database content are not necessarily part of the model's training data.
This is where RAG — Retrieval-Augmented Generation comes in.
Instead of asking an LLM to answer directly, we first retrieve relevant information from our own data and provide that information as context to the model.
In this article, we'll build the foundation of a RAG application using Spring AI.
What is RAG?
RAG stands for:
Retrieval-Augmented Generation
The idea is simple:
User Question
↓
Retrieve Relevant Information
↓
Add Retrieved Context to Prompt
↓
LLM
↓
Generated Answer
For example, imagine we have a company's internal documentation.
A user asks:
What is our refund policy for annual subscriptions?
Instead of expecting the LLM to magically know the answer, our application:
- Searches the company's documents.
- Finds the relevant refund-policy content.
- Adds that content to the prompt.
- Sends the prompt to the LLM.
- Generates an answer grounded in the retrieved information.
This is the core idea behind RAG.
Why Do We Need RAG?
A normal LLM application looks like this:
User
↓
Application
↓
LLM
↓
Answer
The problem?
The LLM only has access to the information available to it.
With RAG, the architecture becomes:
┌──────────────┐
│ Documents │
└──────┬───────┘
↓
Chunking
↓
Embeddings
↓
Vector Database
↑
│
User → Query → Similarity Search
↓
Relevant Context
↓
LLM
↓
Answer
Now the model can work with information from our own knowledge base.
The RAG Pipeline
A production RAG pipeline typically contains these stages:
Documents
↓
Document Loading
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Similarity Search
↓
Relevant Context
↓
Prompt
↓
LLM
↓
Answer
Let's understand each step.
1. Document Loading
First, we need to get our data into the application.
The source could be:
- PDF files
- Markdown files
- HTML pages
- Word documents
- Database records
- APIs
- Knowledge bases
- Internal documentation
Spring AI provides abstractions for working with documents.
A document can be represented using Spring AI's Document abstraction.
Conceptually:
Document document = new Document(
"Spring Boot is a framework for building Java applications..."
);
We now have content that can be processed by our RAG pipeline.
2. Chunking
We shouldn't usually store an entire document as a single vector.
Imagine a 100-page PDF.
A user asks:
How do I configure authentication?
We don't want to retrieve the entire PDF.
Instead, we split the document into smaller pieces called chunks.
For example:
Document
↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
...
Chunk 100
Each chunk represents a smaller piece of knowledge.
A simple example:
Chunk 1:
Introduction to Spring Security
Chunk 2:
Configuring authentication
Chunk 3:
Creating users
Chunk 4:
JWT authentication
Chunk 5:
Role-based authorization
Now when the user asks about JWT authentication, we can retrieve the relevant chunk instead of the entire document.
3. Embeddings
This is where things get interesting.
A computer doesn't understand the semantic meaning of text in the same way humans do.
We need a way to represent text numerically.
That's where embeddings come in.
An embedding model converts text into a vector.
For example:
"How do I configure JWT authentication?"
might become something conceptually like:
[0.12, -0.42, 0.87, 0.31, ...]
The actual vector contains many dimensions.
The important part is:
Semantically similar text produces vectors that are relatively close together in vector space.
For example:
"How can I configure JWT?"
↓
[0.12, 0.81, 0.42, ...]
"JWT authentication configuration"
↓
[0.15, 0.78, 0.45, ...]
These vectors should have high similarity.
4. Vector Database
Now we need somewhere to store these embeddings.
That's where a vector database comes in.
Popular choices include:
- PostgreSQL + pgvector
- Pinecone
- Weaviate
- Qdrant
- Milvus
- Elasticsearch
- Redis
For a Spring Boot application, PostgreSQL with pgvector is an especially interesting option because you can keep your relational data and vector data within the same ecosystem.
Conceptually:
Document Chunk
↓
Embedding Model
↓
Vector
↓
Vector Database
Our database might contain something conceptually like:
ID | Content | Embedding
---|----------------------------|----------------
1 | JWT configuration... | [0.12,...]
2 | OAuth2 configuration... | [0.42,...]
3 | Database configuration... | [0.71,...]
5. Similarity Search
Now suppose the user asks:
How do I configure JWT authentication in Spring Boot?
We first generate an embedding for the question.
User Query
↓
Embedding Model
↓
Query Vector
Then we search the vector database for similar vectors.
Query Vector
↓
Vector Database
↓
Similarity Search
↓
Top K Relevant Chunks
For example:
Result 1 → JWT configuration
Result 2 → Spring Security authentication
Result 3 → SecurityFilterChain configuration
These results become our context.
6. Building the Prompt
Now we combine:
- User question
- Retrieved context
For example:
Context:
Spring Security can be configured using SecurityFilterChain.
JWT authentication can be implemented using a custom
authentication filter...
Question:
How do I configure JWT authentication in Spring Boot?
The LLM receives this information and generates the answer.
This is the Augmented part of Retrieval-Augmented Generation.
7. Sending Context to the LLM
The final flow looks like:
User Question
↓
Embedding
↓
Vector Search
↓
Relevant Documents
↓
Prompt + Context
↓
LLM
↓
Answer
The LLM isn't searching the database itself.
Our application retrieves the information first and provides it to the model.
Building RAG with Spring AI
Now let's look at how Spring AI simplifies this architecture.
A typical Spring AI RAG application contains:
Spring Boot
│
├── Document Reader
│
├── Text Splitter
│
├── Embedding Model
│
├── Vector Store
│
└── Chat Model
The exact model and vector store can be swapped without rewriting the entire application.
That's one of the strengths of Spring AI's abstraction-based approach.
Project Setup
Let's create a Spring Boot project.
We'll need Spring AI dependencies for:
- Chat model
- Embeddings
- Vector store
For example, with Maven:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-bedrock-converse</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
</dependency>
You should use versions compatible with the Spring AI version used by your project.
Configuring the Model
If we're using AWS Bedrock, our application needs AWS credentials and model configuration.
Conceptually:
spring:
ai:
bedrock:
aws:
region: us-east-1
The exact configuration depends on the Spring AI version and Bedrock model you're using.
For production environments, don't hard-code AWS credentials inside application.yml.
Use:
Environment Variables
↓
IAM Roles
↓
AWS Credentials Provider Chain
Whenever possible, prefer IAM roles over static credentials.
Creating a Vector Store
For PostgreSQL + pgvector, the architecture looks like:
Spring Boot
↓
Spring AI VectorStore
↓
PostgreSQL
↓
pgvector
The vector store becomes the bridge between our application and the vector database.
Conceptually:
VectorStore vectorStore;
We can then add documents:
vectorStore.add(documents);
And later search:
List<Document> results =
vectorStore.similaritySearch(
SearchRequest.builder()
.query(question)
.topK(5)
.build()
);
The important abstraction here is:
VectorStore
Our application doesn't need to manually implement vector similarity calculations.
Loading Documents
Let's say we have a PDF containing product documentation.
Spring AI provides document readers that can load different document formats.
Conceptually:
var documents = reader.get();
We then split the documents into chunks.
For example:
TokenTextSplitter splitter = new TokenTextSplitter();
List<Document> chunks =
splitter.apply(documents);
The exact splitter and configuration should be chosen based on your document structure and model context window.
Creating Embeddings
The embedding model converts every chunk into a vector.
Conceptually:
Document Chunk
↓
Embedding Model
↓
Vector
Spring AI handles this through the embedding/vector-store integration.
Then we can store the documents:
vectorStore.add(chunks);
Our ingestion pipeline is now:
PDF
↓
Documents
↓
Chunks
↓
Embeddings
↓
Vector Store
Retrieval
Now let's handle the user's question.
Suppose the user asks:
How does authentication work in our application?
We search the vector store:
List<Document> results =
vectorStore.similaritySearch(
SearchRequest.builder()
.query(question)
.topK(5)
.build()
);
We now have the most relevant pieces of information.
For example:
Result 1
JWT authentication uses...
Result 2
The authentication filter...
Result 3
Security configuration...
Building the RAG Prompt
We can now construct a prompt using the retrieved documents.
For example:
String context = results.stream()
.map(Document::getText)
.collect(Collectors.joining("\n\n"));
String prompt = """
Answer the question using only the provided context.
Context:
%s
Question:
%s
""".formatted(context, question);
Then send it to the chat model.
Conceptually:
ChatResponse response =
chatModel.call(
new Prompt(prompt)
);
And we return:
response.getResult()
.getOutput()
.getText();
That's a basic RAG implementation.
The Complete Flow
Putting everything together:
INGESTION PIPELINE
┌───────────────┐
│ Documents │
└───────┬───────┘
↓
Text Splitting
↓
Embeddings
↓
Vector Store
│
│
│
▼
RETRIEVAL
▲
│
User Question
↓
Embedding
↓
Similarity Search
↓
Top K Chunks
↓
Context
↓
Prompt
↓
LLM
↓
Answer
This is the basic architecture behind many modern knowledge-based AI applications.
Why Chunking Matters
One of the most underestimated parts of RAG is chunking.
Bad chunking can produce bad retrieval.
Imagine this document:
Spring Security
Authentication allows the application to verify
the identity of a user.
Authorization determines whether the authenticated
user has permission to access a resource.
If we split this badly:
Chunk 1:
Authentication allows...
Chunk 2:
the identity of a user. Authorization determines...
Chunk 3:
whether the authenticated user...
we may destroy important semantic relationships.
A better strategy is to preserve meaningful boundaries where possible.
Depending on the data, you may experiment with:
Chunk Size
Overlap
Sentence boundaries
Paragraph boundaries
Markdown headings
Semantic sections
There is no universal chunk size that works for every RAG application.
Why Top-K Matters
When performing similarity search, we usually retrieve the top K results.
For example:
.topK(5)
means:
Return the 5 most relevant chunks.
But bigger isn't always better.
If we retrieve too little:
K = 1
we may miss important context.
If we retrieve too much:
K = 50
we may introduce irrelevant information and increase token usage.
A common approach is to start with a small value and evaluate retrieval quality.
For example:
K = 3
K = 5
K = 10
Then measure which configuration works best for your dataset.
RAG vs Fine-Tuning
A common question is:
Why not fine-tune the model instead?
RAG and fine-tuning solve different problems.
RAG
Best when:
- Knowledge changes frequently
- You have private documents
- You need citations or traceability
- You need to update knowledge without retraining
- You want to retrieve specific information
Fine-Tuning
Useful when you want to change:
- Model behavior
- Output style
- Domain-specific patterns
- Task performance
A useful mental model is:
RAG
→ Give the model the right information.
Fine-tuning
→ Change how the model behaves.
In many production systems, they can also be used together.
RAG Is More Than Vector Search
A basic RAG pipeline is only the beginning.
Production RAG systems often introduce additional stages:
Query
↓
Query Transformation
↓
Hybrid Retrieval
↓
Metadata Filtering
↓
Vector Search
↓
Reranking
↓
Context Compression
↓
Prompt Construction
↓
LLM
You might eventually introduce techniques such as:
- Hybrid search
- Metadata filtering
- Reranking
- HyDE
- Query rewriting
- Self-RAG
- Corrective RAG
- Graph RAG
- Agentic RAG
This is where RAG becomes an engineering discipline rather than simply "put documents into a vector database."
Production Considerations
If you're building RAG for production, don't stop at:
PDF → Vector DB → LLM
You also need to think about:
1. Document ingestion
How frequently are documents updated?
New document
↓
Process
↓
Chunk
↓
Embed
↓
Update Vector Store
2. Metadata
Store useful metadata alongside chunks:
document_id
source
page_number
tenant_id
created_at
updated_at
document_type
This becomes extremely useful for filtering.
For example:
tenant_id = "company-123"
can ensure that users only retrieve documents belonging to their tenant.
3. Access Control
This is critical.
A RAG system must not retrieve documents that the current user isn't authorized to access.
Your retrieval layer should respect application permissions.
User
↓
Authentication
↓
Authorization
↓
Metadata Filters
↓
Retrieval
↓
LLM
Never assume that because the LLM can't "see" a document directly, the document is secure.
4. Observability
Track things such as:
Retrieval latency
Embedding latency
LLM latency
Token usage
Retrieved chunks
Similarity scores
Failure rate
Answer quality
Without observability, debugging RAG becomes extremely difficult.
A Simple Mental Model
If you're new to RAG, remember this:
Embedding Model
=
"Convert meaning into numbers"
Vector Database
=
"Store and search those meanings"
Retriever
=
"Find relevant information"
LLM
=
"Use that information to generate an answer"
Together:
Retrieve → Augment → Generate
That's RAG.
Where Spring AI Fits
Spring AI gives Java developers abstractions around many of these building blocks.
Instead of manually wiring every AI provider and vector database integration, we can work with abstractions such as:
ChatModel
EmbeddingModel
VectorStore
Document
DocumentReader
This allows us to focus more on the application architecture rather than provider-specific implementation details.
And that's particularly useful when building enterprise Java applications where we may want to change:
AWS Bedrock
↓
Another Model Provider
without completely rewriting our application.
Final Architecture
A production-oriented Spring AI RAG application can eventually look like this:
┌───────────────────┐
│ User / App │
└─────────┬─────────┘
↓
Spring Boot API
↓
Query Processing
↓
Retrieval Layer
↓
┌───────────────────┐
│ Vector Store │
│ PostgreSQL │
│ + pgvector │
└─────────┬─────────┘
↓
Relevant Context
↓
Prompt Assembly
↓
┌───────────────────┐
│ Spring AI │
└─────────┬─────────┘
↓
┌───────────────────┐
│ AWS Bedrock │
└─────────┬─────────┘
↓
Answer
And separately:
Documents
↓
Document Reader
↓
Chunking
↓
Embedding Model
↓
PostgreSQL + pgvector
What's Next?
We started this series by exploring how Spring AI can connect Java applications with modern AI models.
With AWS Bedrock, we can access powerful foundation models without managing the underlying model infrastructure.
Now, with RAG, we can take another major step:
We can connect those models to our own data.
The journey now looks like:
LLM
↓
Spring AI
↓
AWS Bedrock
↓
RAG
↓
Vector Database
↓
Our Own Data
But there is another important capability missing.
What if we don't just want the model to answer questions?
What if we want the model to take actions?
For example:
User
↓
AI Agent
↓
Decide what to do
↓
Call a Tool
↓
Execute Action
↓
Return Result
That's where tool calling and AI agents come in.
Next up: Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows.
Key Takeaways
- RAG connects LLMs to external knowledge.
- Documents are split into smaller chunks.
- Embeddings convert text into vectors.
- Vector databases store and retrieve those vectors.
- Similarity search finds relevant information.
- Retrieved context is added to the LLM prompt.
- Spring AI provides abstractions for building these pipelines.
- PostgreSQL + pgvector is a practical option for Java applications.
- Production RAG requires security, metadata filtering, observability, and evaluation.
- RAG is not just vector search — retrieval quality ultimately determines answer quality.
If you're building AI applications with Java and Spring Boot, RAG is one of the most important patterns to understand.
