Introduction
In this tutorial we will build a command-line research assistant that answers questions, remembers context across turns, and calls a mock search tool. It is a practical introduction to how Large Language Models work and how you can productize them with Oxlo.ai.
What you'll need
- Python 3.10 or newer.
- The OpenAI SDK:
pip install openai. - An Oxlo.ai API key from https://portal.oxlo.ai.
Sign up for a free Oxlo.ai account. The free tier includes 60 requests per day, which is enough to build and test this agent. Because Oxlo.ai uses flat per-request pricing, long system prompts and conversation history do not increase your cost. See https://oxlo.ai/pricing for details.
Step 1: Make your first LLM call
First, verify that your environment can reach Oxlo.ai. We will send a single user message to Llama 3.3 70B and print the reply.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "user", "content": "What is a Large Language Model?"},
],
)
print(response.choices[0].message.content)
Step 2: Shape behavior with a system prompt
Raw models return generic text. A system prompt tells the model it is a research assistant and sets rules for tone, length, and tool use. Store it as a constant so you can iterate quickly.
SYSTEM_PROMPT = """You are a concise research assistant.
- Answer in plain English.
- If you need external data, call the search_knowledge_base tool.
- Keep responses under three paragraphs unless asked for detail.
"""
Now send the same question with the system prompt included.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a concise research assistant.
- Answer in plain English.
- If you need external data, call the search_knowledge_base tool.
- Keep responses under three paragraphs unless asked for detail.
"""
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "What is a Large Language Model?"},
],
)
print(response.choices[0].message.content)
Step 3: Add conversation memory
LLMs are stateless. To let the user ask follow-up questions, we maintain a messages list and append each exchange to it.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a concise research assistant.
- Answer in plain English.
- If you need external data, call the search_knowledge_base tool.
- Keep responses under three paragraphs unless asked for detail.
"""
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "What is a Large Language Model?"},
]
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
)
assistant_reply = response.choices[0].message.content
print(assistant_reply)
# Append the exchange to history
messages.append({"role": "assistant", "content": assistant_reply})
messages.append({"role": "user", "content": "How is it different from a search engine?"})
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
)
print(response.choices[0].message.content)
Step 4: Give the agent a tool
Real assistants need data beyond the model's training cutoff. We will register a mock search tool and let the model decide when to call it. The loop checks for a tool call, executes the local Python function, and sends the result back to the model.
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a concise research assistant.
- Answer in plain English.
- If you need external data, call the search_knowledge_base tool.
- Keep responses under three paragraphs unless asked for detail.
"""
def search_knowledge_base(topic: str) -> str:
data = {
"Oxlo.ai": "Oxlo.ai is a developer-first AI inference platform with flat per-request pricing for open-source LLMs.",
"LLM": "An LLM is a neural network trained on vast text to predict the next token.",
}
return data.get(topic, f"No exact entry found for {topic}.")
tools = [
{
"type": "function",
"function": {
"name": "search_knowledge_base",
"description": "Look up a topic in the internal knowledge base.",
"parameters": {
"type": "object",
"properties": {
"topic": {"type": "string", "description": "The topic to search for."}
},
"required": ["topic"],
},
},
}
]
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "What is Oxlo.ai?"},
]
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
tools=tools,
tool_choice="auto",
)
message = response.choices[0].message
# Check if the model wants to call a tool
if message.tool_calls:
tool_call = message.tool_calls[0]
arguments = json.loads(tool_call.function.arguments)
result = search_knowledge_base(**arguments)
messages.append({
"role": "assistant",
"content": message.content or "",
"tool_calls": [
{
"id": tool_call.id,
"type": "function",
"function": {
"name": tool_call.function.name,
"arguments": tool_call.function.arguments,
},
}
],
})
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result,
})
final_response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
)
print(final_response.choices[0].message.content)
else:
print(message.content)
Step 5: Stream the response
Waiting for the full response to return feels slow. We enable streaming so tokens arrive as they are generated. This is especially useful for long explanations.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a concise research assistant.
- Answer in plain English.
- If you need external data, call the search_knowledge_base tool.
- Keep responses under three paragraphs unless asked for detail.
"""
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "Explain how transformers enable LLMs to understand context."},
]
stream = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
stream=True,
)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
print(token, end="", flush=True)
print()
Run it
Here is the complete, runnable script that combines memory, tool use, and streaming. Save it as assistant.py and run python assistant.py.
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a concise research assistant.
- Answer in plain English.
- If you need external data, call the search_knowledge_base tool.
- Keep responses under three paragraphs unless asked for detail.
"""
def search_knowledge_base(topic: str) -> str:
data = {
"Oxlo.ai": "Oxlo.ai is a developer-first AI inference platform with flat per-request pricing for open-source LLMs.",
"LLM": "An LLM is a neural network trained on vast text to predict the next token.",
}
return data.get(topic, f"No exact entry found for {topic}.")
tools = [
{
"type": "function",
"function": {
"name": "search_knowledge_base",
"description": "Look up a topic in the internal knowledge base.",
"parameters": {
"type": "object",
"properties": {
"topic": {"type": "string", "description": "The topic to search for."}
},
"required": ["topic"],
},
},
}
]
def run_assistant():
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "What is Oxlo.ai and how does its pricing work?"},
]
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
tools=tools,
tool_choice="auto",
)
message = response.choices[0].message
if message.tool_calls:
tool_call = message.tool_calls[0]
arguments = json.loads(tool_call.function.arguments)
result = search_knowledge_base(**arguments)
messages.append({
"role": "assistant",
"content": message.content or "",
"tool_calls": [
{
"id": tool_call.id,
"type": "function",
"function": {
"name": tool_call.function.name,
"arguments": tool_call.function.arguments,
},
}
],
})
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result,
})
stream = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
stream=True,
)
print("Assistant: ", end="", flush=True)
for chunk in stream:
token = chunk.choices[0].delta.content
if token:
print(token, end="", flush=True)
print()
else:
print("Assistant:", message.content)
if __name__ == "__main__":
run_assistant()
Example output:
Assistant: Oxlo.ai is a developer-first AI inference platform. Unlike token-based providers, it charges a flat cost per API request regardless of prompt length, which makes it significantly cheaper for long-context and agentic workloads.
Next steps
Swap llama-3.3-70b for qwen-3-32b when you need multilingual reasoning, or kimi-k2.6 for advanced agentic coding and vision. You can also extend the tool suite with real APIs, such as fetching live weather or stock prices, to turn this assistant into a production agent.
