One Open Source Project a Day (No. 166): Needle 2 — A 14 MB On-Device Tool-Calling Model

Introduction

"The 2-bit model you deploy is the model that was trained."

This is the 166th article in the "One Open Source Project a Day" series. Today's project is Needle 2.

On-device deployment of large language models runs into the same three-way constraint every time: parameter count, memory, and latency. To get AI running on a phone or embedded device, you either sacrifice accuracy, speed, or both.

Cactus Compute found a counterintuitive way around this wall: don't build a general model — build only for tool calling.

Needle 2 is a foundation model designed specifically for tool calling, structured extraction, and device control:

• 45M parameters, CQ2-bit quantized
• Single 14 MB binary — no separate model files to manage
• Fixed ~28 MB session RAM — memory does not grow with conversation length
• Matches FunctionGemma 270M (six times the size) on tool-calling benchmarks

9.2k Stars, Apache 2.0, already deployed in production on the Pebble Index Ring and other wearables.

What You Will Learn

• Needle 2's SAN (Simple Attention Network) architecture and its three core design decisions
• Why CQ2-bit quantization has an advantage over post-training quantization
• How confidence gating enables intelligent local/cloud routing
• How to wire up tools with the @needle.tool decorator
• The full LoRA fine-tuning → .cact export pipeline

Prerequisites

• Familiarity with the concept of Function Calling / Tool Use
• Python basics (comfortable with decorators)
• A rough understanding of model quantization (know what INT4/INT8 means)

Project Background

What It Is

Needle 2 is not a scaled-down general-purpose chat model. It is a foundation model designed from scratch specifically for tool-calling tasks.

The question it answers is: if a model only needs to do one thing well — "receive a natural-language query → decide which tool to call → generate correct parameters → return a structured result" — how small can it get?

The answer is 14 MB, 28 MB at runtime.

To put those numbers in context: the average mobile app install is larger than this. 28 MB of RAM is zero pressure on any Android phone in 2026. On a Raspberry Pi 5 it decodes at 500+ tok/s.

Author / Team

• Company: Cactus Compute, Inc. (San Francisco)
• Core members: Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, and several co-founders
• Focus: On-device AI inference startup; the Needle series is its flagship product
• Contact: founders@cactuscompute.com

Project Stats

• ⭐ GitHub Stars: 9,200+
• 🍴 Forks: 592
• 📄 License: Apache 2.0
• 🧪 Paper: arxiv.org/abs/2607.18363
• 🤗 Weights: HuggingFace Cactus-Compute/needle2
• 📦 Install: pip install cactus-needle

Core Features

What Problem It Solves

Needle 2 provides three capabilities — all variations of the same fundamental transformation: natural language → structured output.

Natural-language query
    ↓
Needle 2 inference engine (14 MB binary, 28 MB RAM)
    ↓
┌──────────────┬──────────────────┬─────────────────┐
│  Tool calls  │ Structured JSON  │  Device actions  │
└──────────────┴──────────────────┴─────────────────┘

Every output includes a confidence score that you can threshold to decide whether to handle the request locally or escalate to a cloud model.

Usage Scenarios

1. Offline voice assistant on wearables

   • Pebble Index Ring already uses Needle in production. Voice commands are parsed into API calls locally — no network required.

2. IoT and smart home control

   • Run local voice control on a Raspberry Pi or ESP32. "Turn off the bedroom lights" → turn_off(room="bedroom") — zero cloud latency.

3. Document structured extraction

   • Extract unstructured text (invoices, tables, reports) into strongly-typed JSON. Schema constraints guarantee the output format is correct.

4. On-device mobile AI agent

   • Run a local agent on Android/iOS that handles simple tasks, and routes complex ones to the cloud based on confidence — saves API costs and reduces latency.

5. Spatial computing

   • Decodes at 400–1,500 tok/s on Meta Quest 3S or Apple Vision Pro — fast enough for real-time spatial computing scenarios.

Quick Start

pip install cactus-needle

Tool calling:

import needle

@needle.tool
def get_weather(city: str) -> dict:
    """Get the current weather for a city.

    Args:
        city: The city name to query.
    """
    return {"city": city, "temp_c": 27, "sky": "clear"}

@needle.tool
def send_message(to: str, body: str) -> dict:
    """Send a message to a contact.

    Args:
        to: Recipient name or number.
        body: Message content.
    """
    return {"sent": True}

agent = needle.Needle(tools=[get_weather, send_message])
result = agent.run("What's the weather like in Lagos right now?")
print(result["results"])  # [{"city": "Lagos", "temp_c": 27, "sky": "clear"}]

Structured extraction:

from typing import Annotated
import needle

schema = {
    "invoice_number": str,
    "total_amount": Annotated[float, needle.Field(gt=0)],
    "line_items": list[{"description": str, "qty": int, "unit_price": float}],
}

text = "Invoice #1042, Total: $384.00. 3x Widget @ $128.00"
data = needle.extract(text, schema)
# {"invoice_number": "1042", "total_amount": 384.0, "line_items": [...]}

Confidence gating:

result = agent.complete("Set an alarm for 8am tomorrow")
if result["confidence"] < 0.7:
    # Not confident enough — escalate to cloud
    response = cloud_llm.complete(result["query"])
else:
    # Handle locally
    execute_tool_calls(result["tool_calls"])

Core Features

1. @needle.tool decorator

• Decorate any Python function; the docstring is automatically parsed into a tool schema
• needle.Field constraints (range, regex, enum) are encoded into the JSON Schema and enforced at the grammar level during inference

2. Grammar-constrained decoding

• Structured output is not a soft request ("please output JSON"). The inference engine only permits grammatically legal characters at each token position
• Grammar constraints also allow skipping 98% of the vocabulary projection computation — a major source of speed

3. Confidence score

• Every call returns confidence: the minimum of a posterior calibration head score and the decoding probability
• "The failure mode is escalation, not wrong execution" — better to hand off to the cloud than to execute incorrectly

4. Tool retrieval

• When more than 5 tools are registered, embedding-based retrieval kicks in automatically; only the top 5 most relevant tools per turn enter the context
• "An unselected tool is unreachable, not merely unlikely" — eliminates hallucinated tool calls at the root

5. Engram memory system

• World knowledge is stored in a hashed n-gram table; retrieval costs zero arithmetic operations
• Knowledge and computation are fully decoupled — this is what makes the Hadamard MLP's parameter savings possible

Benchmark Results

Needle 2 matches or beats the strongest competitors on three out of four benchmarks at one-sixth the parameter count — the hallmark of a purpose-built model.

Deep Dive

SAN Architecture: Three Counterintuitive Design Decisions

Needle 2 uses a Simple Attention Network (SAN) instead of a standard Transformer, built around three design choices.

Decision 1: Hadamard MLP instead of a standard FFN

The FFN layer in a standard Transformer accounts for roughly two-thirds of total parameters. SAN replaces dense projections with a fixed Walsh-Hadamard transform — a linear transformation with no learnable parameters — consuming almost none of the parameter budget.

Standard FFN (parameter-heavy):  x → W1 → ReLU → W2 → output
Hadamard MLP (almost parameter-free):  x → H (fixed transform) → output

This is not a capability trade-off. The "knowledge storage" job is simply moved entirely to the Engram memory system.

Decision 2: Engram memory = hashed n-gram table

World knowledge ("Paris is the capital of France," "syntax of list.append()") lives in a hashed n-gram lookup table. Retrieval at inference time costs zero multiplications and zero parameter consumption.

This separation keeps the "computation part" of the model (Attention + Hadamard) extremely lean without reducing knowledge density.

Decision 3: Multi-lane hyper-connections

A 27-layer × 512-wide network uses multi-lane residual connections for more flexible routing than standard residuals — effectively gaining the routing capacity of a wider model without increasing width.

CQ2-bit: Train for Quantization From Day One

Most model quantization is a post-training operation: train at float16/bfloat16, then quantize to INT4 or INT8. This inevitably loses precision — more so at lower bit widths.

Needle 2's Cactus Quants (CQ2-bit) is optimized for 2-bit quantization from the start of training:

Standard pipeline: FP16 training → post-training quantization → INT4 deploy (precision loss)
CQ2 pipeline: CQ2-aware training → CQ2 deploy (training target = deployment target)

The official framing: "The 2-bit model you deploy is the model that was trained." No train-to-deploy precision gap. Quantization is a design goal, not a compromise.

Three Sources of Inference Speed

Source 1: Weights unpacked in vector registers, never decompressed to memory

Traditional quantized models decompress weights from low precision to high precision before each computation step. CQ2-bit weights can be operated on directly within SIMD registers, eliminating the decompression step entirely.

Source 2: Grammar constraints skip 98% of vocabulary projection

Standard LLM generation evaluates softmax over the entire vocabulary (tens of thousands of tokens) at every step. Needle 2's outputs are structured JSON; at any given position, only a tiny number of characters are grammatically legal. The engine can skip the vast majority of candidates before softmax.

Source 3: Automatic CPU instruction-set detection

A single binary includes optimized paths for multiple instruction sets: SDOT (ARMv8.4), NEON (generic ARM), AVX2 (x86), RISC-V Vector Extension, WASM SIMD — the runtime picks the best path automatically.

LoRA Fine-Tuning: From Data to .cact File

Fine-tuning Needle 2 is far simpler than fine-tuning a general-purpose LLM, because the task scope is narrow and well-defined.

Step 1: Prepare data

{"query": "Remind me about the meeting tomorrow at 3pm", "tools": [{"name": "set_reminder", ...}], "answers": [{"name": "set_reminder", "arguments": {"time": "tomorrow 3pm", "message": "meeting"}}]}
{"query": "What's the weather today?", "tools": [{"name": "set_reminder", ...}], "answers": []}

Important: include roughly 1/8 irrelevant examples ("answers": []) to prevent the model from calling a tool on every single input.

Step 2: Data augmentation (optional)

needle generate-data --augment data.jsonl --num-samples 1000

Step 3: LoRA training

# CPU training
needle finetune data.jsonl --epochs 10 --out adapter.pkl

# GPU acceleration
pip install "cactus-needle[gpu]"
needle finetune data.jsonl --epochs 10 --out adapter.pkl

# Apple Silicon
pip install "cactus-needle[metal]"
needle finetune data.jsonl --epochs 10 --out adapter.pkl

Step 4: Export to .cact

# The LoRA adapter is merged into weights at export time
needle build checkpoints/needle2.pkl --lora adapter.pkl --out tuned.cact

Step 5: Load and use

agent = needle.Needle(tools=[my_tools], weights="tuned.cact")

Post-fine-tuning benchmarks show accuracy improvements of 21–58 percentage points, with fine-tuned Needle 2 surpassing DeepSeek V4 Flash on domain-specific tasks — the core advantage of a specialized model trained on a fixed tool set.

Deployment Speed by Device

The RAM footprint is fixed — it does not grow regardless of conversation length. This is the direct result of the 256-token sliding-window KV cache and is a hard requirement for embedded deployment.

Project Links & Resources

Official Resources

• 🌟 GitHub: https://github.com/cactus-compute/needle
• 📚 Website: https://cactuscompute.com/needle
• 🤗 Weights: huggingface.co/Cactus-Compute/needle2
• 📄 Paper: arxiv.org/abs/2607.18363
• 📦 Install: pip install cactus-needle

Related Resources

• Cactus Compute Blog — design rationale for Needle 1
• SAN Architecture Paper — full architectural details

Summary

Key Takeaways

1. Specialized beats general (on fixed tasks): 45M parameters focused on tool calling matches or exceeds models six times larger on three benchmarks
2. CQ2-bit = train-to-deploy: quantization is not a precision trade-off; it was the design target from day one
3. SAN triple combo: Hadamard MLP saves parameters, Engram memory stores knowledge, multi-lane residuals expand routing capacity
4. Confidence gating is calibrated, not guessed: it is a reliable probability signal that can drive local/cloud routing decisions
5. Fixed 28 MB RAM: the 256-token sliding window guarantees memory does not grow with conversation length — a hard requirement for embedded deployment

Who This Is For

• On-device AI / edge computing engineers who need reliable tool-calling capability on resource-constrained hardware
• IoT and smart home developers building offline voice command parsing with zero cloud dependency
• Mobile app developers who want to embed a local agent without relying on cloud APIs
• AI researchers interested in SAN architecture, CQ2-bit quantization, and on-device inference optimization

One-Line Verdict

Needle 2 proves one thing: when you narrow the problem scope enough, 14 MB can beat 270 MB. That is not a trick — it is an architectural philosophy.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage