I’ll show you how to get an ai agent python ollama up and running on your laptop, expose it through FastAPI, test it automatically, and push it to production. You can have a working endpoint in under an hour if you follow the steps carefully.
How do I install and configure Ollama for local LLM inference?
The first thing you need is a running Ollama server. It ships as a single binary for Linux, macOS, and Windows, so there are no heavyweight dependencies.
# Download the latest release (replace with the version you need)
curl -L https://ollama.com/download/ollama-linux-amd64 -o ollama
chmod +x ollama
sudo mv ollama /usr/local/bin/
Start the daemon:
ollama serve &
Ollama defaults to listening on http://127.0.0.1:11434. You can change the bind address with the --host flag or set a custom port with --port. I keep it on the default because my FastAPI container can reach the host network via Docker’s host mode.
Next, pull a model. Ollama supports many open-source LLMs; for a quick start I use phi3 because it fits in 2 GB of RAM.
ollama pull phi3
If you’re on a low-memory box, add --cpu to force CPU inference. It’s slower, but it avoids OOM kills that have bitten me on cheap VPS instances.
Configuration tip: create a ~/.ollama/config.yaml:
host: "0.0.0.0"
port: 11434
log_level: "info"
That way you can restart the service without remembering flags.
How can I build an AI agent in Python using Ollama’s REST API?
Ollama exposes a simple JSON-over-HTTP API. The core endpoint is POST /api/chat. Here’s a minimal wrapper that turns the HTTP call into a Python class.
import httpx
from typing import List, Dict
class OllamaChat:
def __init__(self, model: str = "phi3", base_url: str = "http://127.0.0.1:11434"):
self.base_url = base_url.rstrip("/")
self.model = model
self.client = httpx.Client(timeout=30.0)
def chat(self, messages: List[Dict[str, str]]) -> str:
payload = {
"model": self.model,
"messages": messages,
"stream": False
}
resp = self.client.post(f"{self.base_url}/api/chat", json=payload)
resp.raise_for_status()
return resp.json()["message"]["content"]
The messages list follows the OpenAI chat format (role = system|user|assistant). Because Ollama mimics that schema, you can reuse prompt engineering tricks you already know.
Agent pattern: I like to encapsulate a single “goal” function. For example, a simple calculator agent:
def calculator_agent(question: str) -> str:
wrapper = OllamaChat()
msgs = [
{"role": "system", "content": "You are a reliable calculator. Return only the numeric answer."},
{"role": "user", "content": question}
]
return wrapper.chat(msgs)
When I first built this, I forgot to set stream: False and got a generator back, which broke downstream code that expected a string. Always verify the response shape in a unit test.
How do I integrate the Ollama-powered agent into a FastAPI service?
FastAPI makes wiring an endpoint trivial. Below is a complete main.py that exposes /calculate and forwards the request to the calculator_agent.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Literal
app = FastAPI(title="Ollama AI Agent Service")
class CalcRequest(BaseModel):
expression: str
class CalcResponse(BaseModel):
result: str
@app.post("/calculate", response_model=CalcResponse)
def calculate(req: CalcRequest):
try:
answer = calculator_agent(req.expression)
except httpx.HTTPError as exc:
raise HTTPException(status_code=502, detail=f"Ollama error: {exc}") from exc
return CalcResponse(result=answer)
Run it with:
uvicorn main:app --host 0.0.0.0 --port 8000
If you’re containerizing, keep the Ollama server in a sidecar container and share a Docker network. I once tried to run Ollama inside the same container as FastAPI, but the process died when the container restarted, leaving the API hanging. Separate containers give you independent restart policies.
Production tip: enable uvicorn workers (--workers 4) to utilize multiple CPU cores. Ollama itself is single-threaded per model, so you’ll want a pool of models or a load balancer if you expect high QPS.
How do I automate validation and testing of the agent’s actions?
Testing LLM-driven code is tricky because the output is nondeterministic. I approach it in two layers:
- Prompt contract tests – assert that the response adheres to a JSON schema or a regex pattern.
- Functional integration tests – spin up a real Ollama instance (via Docker) and run end-to-end requests.
Here’s a pytest fixture that starts Ollama in a temporary container:
import subprocess
import time
import pytest
@pytest.fixture(scope="session")
def ollama_server():
proc = subprocess.Popen(["docker", "run", "--rm", "-p", "11434:11434", "ollama/ollama"], stdout=subprocess.PIPE)
# Wait for the health endpoint
for _ in range(10):
try:
httpx.get("http://127.0.0.1:11434/health").raise_for_status()
break
except Exception:
time.sleep(1)
else:
pytest.fail("Ollama did not start in time")
yield
proc.terminate()
And a test that validates the calculator:
def test_calculator_agent(ollama_server):
result = calculator_agent("What is 7 * 8?")
assert result.strip() == "56"
If the model ever drifts (e.g., it starts adding explanations), the test will fail, alerting you before you ship.
When NOT to rely on this: for creative generation (storytelling, code synthesis) you can’t lock down exact output. In those cases, check for presence of required fields rather than exact string equality.
How do I deploy and scale the Ollama agent in production?
Running Ollama on a single machine works for prototypes, but production traffic often exceeds a single CPU’s capacity. My current stack uses three layers:
-
Model shards – run multiple Ollama containers, each loading the same model but pinned to a different CPU core using
taskset. -
Reverse proxy – an Nginx or Traefik instance routes
/api/chatrequests round-robin to the shards. - FastAPI autoscaling – Kubernetes Horizontal Pod Autoscaler (HPA) watches the FastAPI service’s request latency and spins up more pods as needed.
A minimal Kubernetes deployment for Ollama looks like:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ollama
spec:
replicas: 3
selector:
matchLabels:
app: ollama
template:
metadata:
labels:
app: ollama
spec:
containers:
- name: ollama
image: ollama/ollama:latest
ports:
- containerPort: 11434
resources:
limits:
cpu: "2"
memory: "4Gi"
command: ["ollama", "serve", "--host", "0.0.0.0", "--port", "11434"]
Combine that with a Service:
apiVersion: v1
kind: Service
metadata:
name: ollama
spec:
selector:
app: ollama
ports:
- protocol: TCP
port: 11434
targetPort: 11434
Your FastAPI pod just points to http://ollama:11434. If a node runs out of memory, the pod restarts – I’ve added a livenessProbe that hits /health to catch silent OOM kills.
Cost considerations: each model instance reserves RAM for the weights. A 3-B parameter model needs ~6 GB. Running three shards can easily hit 20 GB, so budget accordingly. If you’re on a spot-instance fleet, be prepared for occasional restarts; make your FastAPI client retry with exponential back-off.
When not to scale this way: if you need sub-millisecond latency, Ollama’s CPU inference isn’t enough. In that case, move to a GPU-backed service or a hosted provider that offers hardware acceleration.
FAQ
What language models does Ollama support?
Ollama ships with many open-source models: Llama 3, Mistral, Phi-3, and community-built variants. You pull them with ollama pull <model> and they run on CPU or GPU depending on your hardware.
Can I use the same code with OpenAI’s API?
Yes. The request schema (model, messages, stream) mirrors OpenAI’s chat endpoint, so swapping the base URL and API key converts the wrapper to an OpenAI client. See the related post “Using the OpenAI Agents SDK for Python: A Practical Guide”.
Do I need to restart Ollama after changing the model?
No. After pulling a new model you can start using it immediately by passing the model name to OllamaChat. The server loads the weights lazily on the first request.
How do I monitor Ollama’s performance?
Expose the /metrics endpoint (Prometheus format) by running Ollama with --metrics. Then scrape it with Grafana or Prometheus to watch request latency, CPU usage, and cache hits.
Key Takeaways
- Install Ollama once, pull a lightweight model, and you have a local LLM server ready for HTTP calls.
- Wrap the
/api/chatendpoint in a thin Python class; it gives you a reusable ai agent python ollama building block. - FastAPI integration is a few lines of code; keep Ollama in a sidecar or separate pod for resilience.
- Automated tests that spin up a temporary Ollama container catch prompt drift before it reaches users.
- Scale by sharding model instances behind a reverse proxy and let Kubernetes handle FastAPI pod autoscaling.
- Watch memory footprints; each model instance eats several gigabytes, so plan resources or consider GPU alternatives for low-latency workloads.
With these pieces in place, you can move from a local experiment to a production-grade AI agent service without reinventing the wheel. Happy coding!
