Demystifying the AI Wave: A Backend Engineer's Guide to LLMs, RAG, and Agents

Table of Contents 🗒️

Where it all starts: LLMs
Making LLMs smarter: RAG
Plugging everything in: MCP
The big leap: AI Agents
A tale of two protocols: MCP and A2A
Where does this leave us as engineers?
LangChain: The Backbone of AI Engineering

I've been diving deep into AI lately, trying to demystify this massive wave that's been taking the industry by storm. For a bit of background: I'm a backend software engineer and my sweet spot is Java. I absolutely love solving complex system design problems using object-oriented programming.

I first dipped my toes into AI back in early 2023 when ChatGPT went viral. Back then, I used it like everyone else; as a handy chatbot for quick answers. But recently, I realized it's time to move past just using AI and start actually building with it. So I did what any curious engineer would do: I went down the rabbit hole. Countless blog posts, YouTube videos, and Google's free AI Agents intensive course later, I finally feel like things are starting to click.

Now that the dust has settled, I've built a solid mental model of how AI, LLMs, and agents fit together. In this post, I want to share that roadmap and give you a clear, high-level overview of the core concepts you need to know to start building too. Think of this as your cheat sheet; a quick ramp-up for software engineers that cuts through the noise and gives you direction without the overwhelm.

Where it all starts: LLMs 🤖

Our first real introduction to modern AI was through Large Language Models, or LLMs. On the surface it seems simple: you type a question, and the LLM spits out an answer. Under the hood though, it pulls off this magic using two core pillars: Transformers and Vector Databases.

When you first start looking into AI, it's really easy to feel overwhelmed. You might think you need to familiarize yourself with traditional machine learning concepts like

Supervised and unsupervised learning
K-Means clustering
Q-learning
Principal Component Analysis (PCA)
Logistic regression and many more...

But let me cut down the noise for you: as a software engineer, you don’t need to know all of that just yet. While you can certainly learn those traditional ML algorithms later if you're curious, they aren't prerequisites for building with generative AI today.

1. Vector Embeddings: The AI's Language 💬

Computers don't get words, but they love math. Vector embeddings are the ultimate translator. They take human text and convert it into a massive string of numbers (a vector).

The trick here is that these numbers represent meaning. Think of it like a giant, multi-dimensional map. Words that mean similar things get placed right next to each other on the map.

In practice: "King" and "Queen" will have coordinates right next to each other, while "Apple" will be parked miles away.

2. Transformers: The Brain 🧠

If embeddings are the vocabulary, the Transformer is the brain doing the reading. It’s the game-changing neural network architecture behind every major LLM today.

Old-school AI used to read sentences sequentially—one word at a time—which meant it totally lost the plot by the end of a long paragraph. Transformers process the entire text block all at once. Using something called the Self-Attention Mechanism, the model instantly links words together to figure out context, no matter how far apart they are in the sentence.

In practice: If you say, "The bank of the river was muddy, so I couldn't withdraw cash," the Transformer instantly hooks the word "bank" to "river" and "withdraw" at the exact same time, perfectly sorting out the double meaning.

3. Vector Databases: The Filing Cabinet 🗄️

Standard SQL databases are great for exact matches, but they are completely blind to nuance. If you search a SQL database for "refund policy," it won't find a document that says "cashback guidelines" because the characters don't match.

A Vector Database (like pgvector or Pinecone) is a specialized filing cabinet built to store and search those numeric coordinates we talked about. Instead of looking for exact letters, it calculates geometric distance. It takes your prompt, turns it into a coordinate, and pulls the files that are physically closest to it in semantic meaning.

The TL;DR Pipeline:

When you hit enter on a prompt, they all high-five:

Your text becomes a Vector Embedding (coordinates).
The Vector Database finds the closest matching data coordinates.
The Transformer eats your prompt + that data all at once, handles the context, and spits out the perfect answer.

If you want a fantastic visual breakdown of how transformers work under the hood, I highly recommend this video and some of the ones linked below:

Making LLMs smarter: RAG

Now that we've pulled back the curtain on Transformers and vector databases, let's talk about the next logical step: RAG, or Retrieval-Augmented Generation.

Out of the box, foundational LLMs, namely, Google Gemini, OpenAI GPT-4o, Anthropic Claude, or Meta Llama 3, only know what they were trained on. Ask them about anything outside that, like recent news or your company's internal documents, and they'll either admit they don't know or worse just hallucinate something.

RAG solves this. Instead of forcing the LLM to rely purely on its memory, RAG lets the model pull in real-time information from external sources before it responds. It's the difference between a colleague who only remembers what they studied in school versus one who can actually Google things before answering you.

Plugging everything in: MCP

So RAG lets LLMs reach out for external data, but how does a model connect to all these different sources without developers writing custom integrations every single time? That's where MCP, the Model Context Protocol, comes in.

Introduced by Anthropic as an open-source standard, MCP is basically the USB-C port of AI. Just like HTTP standardized how browsers talk to servers, MCP standardizes how AI models and agents securely fetch data from tools, databases, and file systems. It works through a simple client-server setup:

MCP Clients

The AI apps or agents (think Claude Desktop, Cursor, or ChatGPT) that need external context or want to trigger an action.

MCP Servers

Lightweight programs that connect to specific data sources (like GitHub, Google Drive, or a Slack workspace) and expose that data to the client.

Put RAG and MCP together, and you've gone from a chatbot that only knows what it was trained on to a connected assistant that can work with real-world, real-time data. Pretty powerful upgrade!

The big leap: AI Agents

This is where things get really exciting. To understand why agents are taking the industry by storm, you first need to understand what actually separates an agent from a plain LLM:

An LLM is the brain. It's a powerful model trained on massive amounts of data, but it's fundamentally reactive. You give it a prompt, it processes it, it gives you a response. That's it. No planning, no follow-through.
An AI Agent is the worker. An agent wraps that same LLM in a loop of planning, memory, and tools. It's proactive and goal-oriented. You hand it a complex objective and it figures out how to get there on its own.

A fully realized AI Agent pulls together four things to make that happen:

The Brain (LLM) — The core engine for reasoning and understanding language.
Planning & Reflection — Using patterns like ReAct (Reason + Act), the agent doesn't just answer immediately. It thinks out loud: "To accomplish this, I need to check X first, then evaluate Y, and if that fails, try Z."
Memory — Short-term memory to track what it's already done in the current session, and long-term memory via vector databases to remember preferences or past interactions across sessions.
Tools — This is where MCP or traditional APIs come in. Tools give the agent hands. It can write code, query a database, read a file, send an email — whatever it takes to execute the plan.

We ended up with agents because LLMs, as brilliant as they are, are kind of helpless on their own. Give them memory and a toolkit, and suddenly they can coordinate and knock out complex, multi-step tasks that no single LLM could accomplish on its own.

A Tale of Two Protocols: MCP and A2A

1. MCP: The Vertical Layer (Agent-to-Tool)

As we covered, Anthropic’s Model Context Protocol (MCP) handles how a single agent talks down to its environment. It connects the model vertically to your infrastructure, giving it read/write access to internal SQL databases, filesystem resources, or internal company APIs. MCP is about giving an isolated brain a set of hands to touch data.

2. A2A: The Horizontal Layer (Agent-to-Agent)

Originally introduced by Google, the Agent-to-Agent (A2A) Protocol handles how agents talk horizontally to each other. In complex enterprise systems, you don't build one massive, monolithic agent that knows how to do everything. Instead, you build a network of micro-agents: a coding agent, a billing agent, and a DevOps agent.

A2A defines how these independent nodes discover each other across a network using cryptographic "Agent Cards" (JSON manifests that advertise an agent's specific skills and authentication requirements). Using A2A, a primary agent can securely negotiate, delegate sub-tasks, and stream status updates to another agent across organizational boundaries—even if one is built on LangChain and the other is built on an entirely different framework like CrewAI etc.

The Architectural Takeaway

They aren't competitors; they are a complementary stack. Think of MCP as the internal bus inside a computer linking the CPU to the hard drive, and A2A as the internet protocol (like HTTP) allowing completely separate computers to collaborate.

As a backend engineer, this architecture should feel incredibly familiar. We are essentially watching the wild west of AI reshape itself into a standard, decoupled microservices architecture.

Where does this leave us as engineers?

This is exactly where the industry is heading. Our role as software engineers is shifting from writing rigid, deterministic code to building these dynamic agentic workflows. And the good news? You don't have to build everything from scratch.

The complex coordination loops, retry logic, and state management have already been abstracted away by solid frameworks and the most important one to know about is LangChain.

LangChain: The Backbone of AI Engineering

LangChain is essentially the backbone of the modern AI engineering ecosystem. At its core, it's an open-source framework designed to make building LLM-powered applications and agents dramatically simpler. Instead of manually wiring together your LLM calls, memory, tools, and data sources, LangChain gives you modular, composable building blocks that snap together cleanly. Think of it like Spring Boot, but for AI. It handles the plumbing so you can focus on the logic.

Some of the key things LangChain abstracts away for you:

LLM integrations — Swap between OpenAI, Anthropic, Google Gemini, or any other provider with minimal code changes. No vendor lock-in.
Chains — The ability to link multiple LLM calls or steps together in sequence, where the output of one step feeds into the next.
Memory management — Built-in support for both short-term conversational memory and long-term vector-backed memory, without you having to wire it up yourself.
Tool & agent support — Easily equip your agent with tools (web search, code execution, API calls) and define how it reasons and plans using patterns like ReAct.
RAG pipelines — LangChain has first-class support for document loading, chunking, embedding, and retrieval, making it straightforward to build a RAG system on top of your own data.

Tip: LangChain also comes with LangSmith, an observability and debugging platform that lets you trace exactly what your agent is doing at every step, which becomes invaluable the moment your agent starts doing something unexpected.

If you're in the Python world, LangChain is the clear go-to and has the largest community and ecosystem around it. But if you're a Java backend developer like me! someone who lives in Spring and loves OOP; frameworks like Spring AI and LangChain4j bring these same ideas into the Java ecosystem, letting you spin up fully functioning, production ready agents using the design patterns you already know and love.

This covers the bare-bones of modern AI. Below are some useful videos and resources if you're interested in learning more.