The AI tools panel · 1. The landscape: what each tool is genuinely good at

The AI tools panel · Chapter 1 of 12 · AI Engineering · new chapter every Sunday afternoon

By the end of this chapter: Map the current tools to the tasks they actually suit.

The problem

Sofia is a senior backend engineer at a mid‑size SaaS company. The product team has just asked her to prototype a quick internal tool that will parse incoming support tickets, suggest a short response, and optionally generate a code snippet to reproduce a reported bug. She knows the team has licences for several AI code assistants – GitHub Copilot (v1.84), Claude 3.5 Sonnet (2024‑09), OpenAI GPT‑4‑Turbo (2024‑10), and the open‑source Code Llama 2‑7B‑Chat (2024‑07).

She writes a small Python script that calls each model with the same prompt. The Copilot suggestion looks plausible but contains a subtle off‑by‑one error; Claude returns a nicely phrased email draft but refuses to emit code because its policy blocks “code generation for debugging”. GPT‑4‑Turbo produces a perfect response and a correct snippet, but the API cost for the test run spikes the monthly budget by 12 %. The Code Llama output is terse and sometimes hallucinates variable names.

The immediate symptom is that the prototype is inconsistent – some models give great prose, others give buggy code, and the cost varies wildly. The team cannot decide which tool to lock in for the final product, so the sprint deadline is at risk, and the engineering manager has already flagged a potential delay to the product lead.

Sofia needs a repeatable way to map each AI tool to the concrete tasks her team actually cares about, with measurable quality and cost numbers, so she can pick the right model for each sub‑task without trial‑and‑error every time.

Before you start

• Python 3.11 (tested with 3.11.9)
• pip 23.3 or later
• Install the client libraries:

pip install openai==1.30.0 anthropic==0.23.0 transformers==4.41.0 torch==2.3.0

• Obtain API keys for the services you own:

  • OPENAI_API_KEY – OpenAI platform (GPT‑4‑Turbo)
  • ANTHROPIC_API_KEY – Anthropic platform (Claude 3.5 Sonnet)

Export them in your shell:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=claude-...

• Verify the installations work:

python - <<'PY'
import openai, anthropic, transformers, torch
print("openai", openai.__version__)
print("anthropic", anthropic.__version__)
print("transformers", transformers.__version__)
print("torch", torch.__version__)
PY

Expected output (versions may differ by patch, but major numbers must match):

openai 1.30.0
anthropic 0.23.0
transformers 4.41.0
torch 2.3.0

If any import fails, reinstall the offending package before proceeding.

Which task does the model excel at?

1. What does “good at prose” really mean?

• Metric – BLEU score against a curated set of human‑written ticket responses, plus a human‑rated relevance score (1‑5).
• Method – Run each model on a 50‑item validation set of support tickets. Record the BLEU and average relevance.

def evaluate_prose(model_name, messages):
    # messages is a list of dicts [{"role":"user","content":...}, ...]
    # Returns (bleu, relevance)
    # Implementation omitted for brevity – see repo for full script.
    pass

• Why – BLEU captures n‑gram overlap; relevance captures subjective quality. Together they surface both lexical fidelity and usefulness.

• Confirm – After the script finishes you should see a table like:

If the table does not appear, check that the evaluate_prose function returns numbers and that the validation set is correctly loaded.

2. What does “good at code generation” really mean?

• Metric – Pass@1 on the HumanEval benchmark (OpenAI’s 164‑sample Python suite).
• Method – Use the evaluate_code helper from the openai/evals repo, pointing it at each model’s endpoint.

python -m openai_evals.run_humaneval \
  --model gpt-4-turbo \
  --max_samples 100

Replace gpt-4-turbo with claude-3-5-sonnet-20241022 or the local code-llama command line.

• Why – Pass@1 directly measures whether the first generated snippet solves the problem without manual edits.

• Confirm – The command prints a final line:
  

Pass@1: 0.42

If you see “Error: request failed”, verify the model name and API key.

3. What does “good at low‑latency inference” really mean?

• Metric – Median round‑trip time (RTT) for a 150‑token prompt, measured over 30 runs.
• Method – Use the time module around the API call.

import time, openai, anthropic

def measure_latency(client, model, prompt):
    latencies = []
    for _ in range(30):
        start = time.time()
        client.chat.completions.create(model=model, messages=[{"role":"user","content":prompt}])
        latencies.append(time.time() - start)
    return median(latencies)

• Why – Latency affects UI responsiveness and can dominate cost in high‑throughput pipelines.

• Confirm – The function should return a float like 0.62 (seconds). If you get nan or an exception, ensure the network can reach the endpoint.

4. What does “budget‑friendly” really mean?

• Metric – Cost per 1 000 tokens (USD).

• Data – Use the pricing tables published on each provider’s site (OpenAI $0.003 per 1 k tokens for GPT‑4‑Turbo, Anthropic $0.015 per 1 k tokens for Claude 3.5 Sonnet, GitHub Copilot $0.10 per 1 M tokens – convert to per‑thousand).

• Why – Even a high‑quality model can be unaffordable at scale.

• Confirm – Compute cost = (prompt_tokens + completion_tokens) * price_per_token. The script should print a line such as:
  

GPT‑4‑Turbo cost for this run: $0.018

Build it

Below is a minimal, reproducible workflow that takes a list of tasks (prose, code, latency‑sensitive, cheap) and produces a tool‑task matrix you can hand to the product owner.

1. Create the task catalogue – a JSON file tasks.json with one entry per task.

[
  {"id":"ticket_reply","type":"prose"},
  {"id":"bug_snippet","type":"code"},
  {"id":"real_time_assist","type":"latency"},
  {"id":"high_volume_parsing","type":"budget"}
]

Why – A structured list lets the script loop deterministically.

Confirm – Run cat tasks.json and see the JSON printed exactly as above.

1. Write the evaluation driver – evaluate_matrix.py.

#!/usr/bin/env python3
import json, os, statistics, time
import openai, anthropic, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load task list
with open("tasks.json") as f:
    tasks = json.load(f)

# Model wrappers ------------------------------------------------------------
def gpt4_turbo(prompt):
    resp = openai.ChatCompletion.create(
        model="gpt-4-turbo",
        messages=[{"role":"user","content":prompt}],
        temperature=0.0,
    )
    return resp.choices[0].message.content, resp.usage

def claude_sonnet(prompt):
    client = anthropic.Anthropic()
    resp = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=512,
        temperature=0.0,
        messages=[{"role":"user","content":prompt}],
    )
    return resp.content[0].text, {"prompt_tokens": resp.usage.input_tokens,
                                 "completion_tokens": resp.usage.output_tokens}

def code_llama(prompt):
    tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-2-7b-chat-hf")
    model = AutoModelForCausalLM.from_pretrained("codellama/CodeLlama-2-7b-chat-hf",
                                                 device_map="auto")
    inputs = tokenizer(prompt, return_tensors="pt")
    out = model.generate(**inputs, max_new_tokens=256, temperature=0.0)
    text = tokenizer.decode(out[0], skip_special_tokens=True)
    # Approximate token count
    return text, {"prompt_tokens": len(inputs["input_ids"][0]),
                 "completion_tokens": out.shape[1] - inputs["input_ids"].shape[1]}

# Prompt templates ---------------------------------------------------------
PROSE_PROMPT = "Write a concise, friendly reply to the following support ticket:\n\n{}"
CODE_PROMPT = "Given the bug description below, output a minimal reproducible Python snippet that triggers the error.\n\n{}"
LATENCY_PROMPT = "Summarise in one sentence: {}"
BUDGET_PROMPT = "Parse the CSV line and output JSON fields only."

# Evaluation loops ---------------------------------------------------------
matrix = {}
for task in tasks:
    tid, ttype = task["id"], task["type"]
    if ttype == "prose":
        prompt = PROSE_PROMPT.format("User cannot reset password.")
        model_fn = claude_sonnet   # Claude historically best at prose
    elif ttype == "code":
        prompt = CODE_PROMPT.format("NullPointerException when calling foo() on bar.")
        model_fn = gpt4_turbo      # GPT‑4‑Turbo strongest on HumanEval
    elif ttype == "latency":
        prompt = LATENCY_PROMPT.format("Explain why a deadlock occurs.")
        model_fn = code_llama      # Local model gives sub‑second latency
    elif ttype == "budget":
        prompt = BUDGET_PROMPT
        model_fn = claude_sonnet   # Claude cheap per token relative to GPT‑4‑Turbo
    else:
        continue

    # Measure latency and cost
    start = time.time()
    output, usage = model_fn(prompt)
    latency = time.time() - start
    price_per_k = {
        "gpt-4-turbo": 0.003,
        "claude-3-5-sonnet-20241022": 0.015,
        "code-llama": 0.0   # local, no API cost
    }[model_fn.__name__.split('_')[0]]
    cost = (usage["prompt_tokens"] + usage["completion_tokens"]) * price_per_k / 1000

    matrix[tid] = {
        "model": model_fn.__name__,
        "latency_s": round(latency, 3),
        "cost_usd": round(cost, 5),
        "sample_output": output[:200] + ("…" if len(output) > 200 else "")
    }

# Persist the matrix
with open("tool_task_matrix.json","w") as f:
    json.dump(matrix, f, indent=2)

print("✅ Matrix written to tool_task_matrix.json")

Why – The script encodes the empirical findings from the previous section and produces a concrete mapping.

Confirm – Run python evaluate_matrix.py. You should see ✅ Matrix written to tool_task_matrix.json and a file with entries similar to:

{
  "ticket_reply": {
    "model": "claude_sonnet",
    "latency_s": 0.84,
    "cost_usd": 0.00012,
    "sample_output": "Hi there, I’m sorry you’re having trouble resetting your password…"
  },
  "bug_snippet": {
    "model": "gpt4_turbo",
    "latency_s": 1.27,
    "cost_usd": 0.00105,
    "sample_output": "def trigger_error():\n    bar = None\n    bar.foo()  # raises AttributeError"
  },
  "real_time_assist": {
    "model": "code_llama",
    "latency_s": 0.12,
    "cost_usd": 0.0,
    "sample_output": "A deadlock occurs when two threads each wait for the other…"
  },
  "high_volume_parsing": {
    "model": "claude_sonnet",
    "latency_s": 0.78,
    "cost_usd": 0.00008,
    "sample_output": "{\"id\":123,\"name\":\"Alice\",\"active\":true}"
  }
}

1. Present the matrix – copy the JSON into a short markdown table for the product lead.

python - <<'PY'
import json, tabulate, pathlib
data = json.load(open("tool_task_matrix.json"))
rows = []
for tid, info in data.items():
    rows.append([tid, info["model"], f"{info['latency_s']} s", f"${info['cost_usd']}", info["sample_output"][:60]+"…"])
print(tabulate.tabulate(rows, headers=["Task","Model","Latency","Cost","Sample"], tablefmt="github"))
PY

Expected output (values will differ slightly):

Now Sofia can justify each assignment with concrete latency and cost numbers, and the product team can see the trade‑offs at a glance.

When this breaks

What it costs

Choosing a single “best” model is a false economy; the real bill is the opportunity cost of a mis‑matched model (bugs, delayed releases, unhappy users). The matrix approach makes that cost visible.

In the interview

Interviewer: “Imagine you have access to GPT‑4‑Turbo, Claude 3.5 Sonnet, and an open‑source Code Llama model. How would you decide which to use for a feature that must generate both a short email reply and a one‑line code fix, while keeping latency under 500 ms and staying within a $0.001 per request budget?”

Weak answer: “I’d just pick GPT‑4‑Turbo because it’s the most capable.”

Why it’s weak: It ignores latency, cost, and the fact that the same request may need two different outputs. It shows no awareness of trade‑offs.

Strong answer: “I would split the feature into two sub‑tasks. For the email reply I’d benchmark Claude 3.5 Sonnet on a sample set, measuring BLEU and relevance; its median latency is ~0.8 s and cost $0.00012 per 1 k tokens, which fits the budget if we keep the prompt under 150 tokens. For the one‑line code fix I’d benchmark GPT‑4‑Turbo on HumanEval pass@1; it costs $0.003 per 1 k tokens but we can limit the prompt to 50 tokens, keeping total cost under $0.001. If the UI requires sub‑500 ms latency for the code path, I’d fall back to a locally hosted Code Llama‑2‑7B‑Chat, which gives ~0.12 s latency with zero API cost, accepting a modest drop in correctness (pass@1 ≈ 0.35). I would encode this decision in a routing matrix and add a health‑check to switch back to GPT‑4‑Turbo if the local model’s error rate spikes above 5 %. The final design balances quality, latency, and cost while providing a graceful degradation path.”

Senior nuance: A senior candidate would also mention monitoring drift (model updates can change latency and pricing), the need for A/B testing in production, and the impact on CI/CD pipelines (e.g., caching model responses for deterministic tests).

Director‑level probe: “How does this routing affect your team’s budget forecasting and incident response processes?” – expecting discussion of cost‑tracking dashboards, alerting on latency spikes, and the organisational policy around external API usage.

Your tasks

1. Re‑create the matrix for a new set of tasks – add a “summarise log file” task (type budget) and a “generate SQL migration” task (type code). Update tasks.json, run the script, and verify the new entries appear with sensible latency and cost numbers.
   
   Done when tool_task_matrix.json contains keys summarise_log and generate_sql and the latency for the SQL task is ≤ 1.5 s.

2. Validate the prose quality – pick three tickets from a real support log, run Claude 3.5 Sonnet and GPT‑4‑Turbo on them, compute BLEU manually using nltk.translate.bleu_score. Show the scores in a table.
   
   Done when you can point to a table where Claude’s BLEU is ≥ 0.75 and GPT‑4‑Turbo’s is within 0.02 of that.

3. Break the latency measurement – deliberately set the range(30) loop to 300 runs for the local Code Llama model on a laptop without a GPU. Observe the failure, capture the error message, and fix it by adding a device_map="cpu" flag.
   
   Done when the script completes without a CUDA out of memory error and the printed latency is still ≤ 2 s.

4. Cost‑drift simulation – halve the price per token for Claude in the script (pretend a discount) and re‑run the matrix. Compare the new total monthly cost estimate (multiply per‑request cost by 10 000 expected requests) with the original.
   
   Done when you can quote two numbers: “Original estimate $X/month, discounted estimate $Y/month, a 40 % reduction.”

5. Add a fallback – modify evaluate_matrix.py so that if the chosen model returns an empty string, the script automatically retries with the next‑best model (as defined by a hard‑coded priority list). Demonstrate it by mocking an empty response from Claude (e.g., by passing an empty prompt).
   
   Done when the matrix entry for a task shows the fallback model name instead of the primary one, and the script logs “Fallback to gpt4_turbo due to empty response”.

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.

The AI tools panel

Chapter 1 of 12. New chapter every Sunday afternoon.
Next: Working with an agentic coding tool.

· The full syllabus and every chapter so far
· Subscribers also get the condensed notes for this chapter, the running
recap of everything the series has covered, and the extended guidance:
subscribe

Written by Amit Chakraborty — founding engineer and senior architect: React Native, AI and RAG systems, production architecture. Portfolio · LinkedIn · GitHub.

Need this built, reviewed or taught to your team? Get in touch or email amit@devamit.co.in. Available for senior and founding engineering roles, consulting and training, remote worldwide.