Teams often debate whether a new project needs a classical ML pipeline and a labeling team or just a prompt and an LLM. We are going to build a Task Router agent that settles that debate automatically. It compares the problem against a live traditional ML baseline, then uses an LLM hosted on Oxlo.ai to explain which paradigm fits and why, saving engineering hours and preventing over-engineering.
What you'll need
- Python 3.10+
pip install openai scikit-learn pandas- An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Set up the Oxlo.ai client
I use the OpenAI SDK because Oxlo.ai exposes a fully compatible endpoint. Pointing the client at Oxlo.ai lets me call Llama 3.3 70B with no extra boilerplate.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)Step 2: Train a traditional ML baseline
To make the comparison concrete, I trained a tiny RandomForest on six support tickets. This is the kind of structured, tabular text task where classic ML shines. We will feed its prediction into the LLM as context so the router has something to reason against.
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
# A toy dataset: support tickets with urgency labels
data = [
("reset my password", "low"),
("server is down", "high"),
("need invoice copy", "low"),
("data breach suspected", "high"),
("update billing address", "low"),
("API returns 500 errors", "high"),
]
texts, labels = zip(*data)
# Traditional ML pipeline: TF-IDF + RandomForest
clf = make_pipeline(TfidfVectorizer(), RandomForestClassifier(random_state=42))
clf.fit(texts, labels)
test_input = "production database unreachable"
ml_prediction = clf.predict([test_input])[0]
print(f"ML baseline prediction for '{test_input}': {ml_prediction}")Step 3: Define the router system prompt
The system prompt is the only logic the LLM sees. I keep it strict: output JSON, pick traditional ML for structured data with labels, and pick an LLM for reasoning or generation. This prompt is the heart of the agent.
SYSTEM_PROMPT = """You are an ML architecture advisor.
Your job is to read a task description and decide whether it is best solved by a traditional machine learning model (like logistic regression, random forests, or gradient boosting) or by a large language model.
Rules:
- Recommend traditional ML for structured data, clear input features, numeric tabular data, or well-defined classification with many labeled examples.
- Recommend an LLM for natural language understanding, reasoning, multi-step agentic workflows, few-shot adaptation, or tasks requiring open-ended generation.
- Explain your reasoning in one concise paragraph.
- Output exactly one JSON object with keys: "recommendation" (either "traditional_ml" or "llm"), "confidence" (low, medium, high), and "reasoning" (string)."""Step 4: Call Oxlo.ai to reason about the task
Now we wire the prompt to Llama 3.3 70B through Oxlo.ai. I use Oxlo.ai here because the flat per-request pricing (see https://oxlo.ai/pricing) means I can iterate on the prompt and run many evaluations without watching token meters spin up. The function sends the user task plus the ML baseline context and returns a structured recommendation.
import json
def route_task(task_description, ml_baseline_context):
user_message = f"""Task description:
{task_description}
Traditional ML baseline context:
{ml_baseline_context}
Based on the rules above, recommend the right approach and return valid JSON."""
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
temperature=0.2,
)
content = response.choices[0].message.content.strip()
if content.startswith("
```"):
content = content.split("```
")[1]
if content.startswith("json"):
content = content[4:]
return json.loads(content.strip())Step 5: Run a side-by-side comparison
Finally, we build a small harness that feeds the same business problem to both the scikit-learn pipeline and the Oxlo.ai router. Seeing the two outputs next to each other makes the difference between statistical pattern matching and explicit reasoning obvious.
def compare_approaches(task_description, sample_input=None):
# Run the traditional ML side
if sample_input:
pred = clf.predict([sample_input])[0]
ml_context = (
f"Trained on 6 support tickets. "
f"Sample input '{sample_input}' classified as urgency='{pred}'."
)
else:
ml_context = "No structured training data available for this task."
# Run the LLM router side via Oxlo.ai
result = route_task(task_description, ml_context)
print("=== Traditional ML ===")
print(ml_context)
print("\n=== LLM Router (Oxlo.ai) ===")
print(json.dumps(result, indent=2))
print()Run it
We will test two scenarios. The first is a classic classification job where traditional ML should win. The second is an open-ended generation task where only an LLM makes sense.
if __name__ == "__main__":
# Scenario A: structured classification
compare_approaches(
task_description="Classify customer support tickets by urgency using historical labels.",
sample_input="production database unreachable"
)
# Scenario B: open-ended reasoning and generation
compare_approaches(
task_description="Draft a personalized apology email to a Fortune 500 client after a 12-hour outage, referencing their SLA and offering remediation.",
sample_input=None
)Example output:
=== Traditional ML ===
Trained on 6 support tickets. Sample input 'production database unreachable' classified as urgency='high'.
=== LLM Router (Oxlo.ai) ===
{
"recommendation": "traditional_ml",
"confidence": "high",
"reasoning": "This is a well-defined text classification task with discrete labels and historical examples. A RandomForest with TF-IDF features can learn keyword patterns efficiently without the cost or complexity of an LLM."
}
=== Traditional ML ===
No structured training data available for this task.
=== LLM Router (Oxlo.ai) ===
{
"recommendation": "llm",
"confidence": "high",
"reasoning": "Drafting a context-aware, legally sensitive apology email requires natural language generation, reasoning over an SLA document, and tone adaptation. These open-ended capabilities are outside the scope of traditional ML classifiers."
}Wrap-up
That is the full agent. It is not perfect, but it is useful. Two concrete next steps: integrate Oxlo.ai function calling so the router can automatically scaffold a scikit-learn notebook when it picks traditional ML, and add a streaming response so users see the reasoning appear token by token. Both are straightforward because Oxlo.ai supports the standard OpenAI SDK patterns out of the box.
