I built a robot arm that understands Hinglish — on a Raspberry Pi, with no LLM

Originally published on Medium.

How a 322M-parameter "decision model" called Laya, two small speech recognisers and a 50 Hz motion loop turned "haath hilao" into a wave — offline, in about a tenth of a second.

Most people I work with don't speak English or Hindi. They speak both at once: "pen uthao quickly", "haath hilao yaar", "धीरे से झुको". I wanted a robot I could talk to that way. And I wanted it to run on a Raspberry Pi with no internet, because a robot that stops working when the Wi-Fi drops isn't much of a robot.

The obvious design in 2026 is "put a language model in the loop". I tried the other direction, and I think it's the better one for this kind of job. Here's what I built, what the numbers look like, and where it still fails.

Code, training script, evaluation and demo video: github.com/dockndevai/pi-meeting-assistant (MIT).

"wave hello to everyone" → WAVE, 100%, 122 ms. The left panel is the simulator drawing the arm from its live joint angles.

The pipeline, end to end

microphone ─► speech-to-text (Vosk, on the Pi)
                    │  Indian-English model + Hindi model, both on every utterance
                    ▼
            Laya: "which gesture does this ask for?  how fast?"
                    │  one forward pass, calibrated confidence, "none" if it's just talk
                    ▼
            motion engine (50 Hz, eased keyframes)
                    │
                    ▼
            PCA9685 servo board → 4 servos        (or the on-screen simulator)

Nothing in that chain leaves the device. No API keys, no cloud, no wake word.

Why not an LLM?

Three reasons, in order of how much they mattered.

1. A robot should not be able to hallucinate an action. A generative model asked to output gesture("wave") can also output gesture("salute"), gesture("wave", repeat=200), or a paragraph of JSON-ish text that doesn't parse. You end up writing a validation layer to protect the servos from the model. With a model that can only choose from a list, that layer is the model.

2. Latency on a Pi. A small LLM on a Pi 5 CPU produces a few tokens per second. Even a one-word answer, after the prompt is processed, is seconds away. For "stop!" that's too slow to feel like control.

3. Confidence you can act on. I wanted a rule like "if you're not sure, don't move." For that, the model's confidence has to actually mean something.

Laya: a model that decides instead of generating

Laya is a non-autoregressive "System 1" decision engine. You give it a piece of text (the state) and a set of typed questions, and it answers all of them in a single encoder forward pass:

• choice — pick one option from a list you define
• score — pick a level on an ordinal scale
• yes/no — a calibrated probability

It's built on ModernBERT / mmBERT encoders (300–420M parameters), ships three checkpoints, and is trained with proper scoring rules so the confidences are calibrated rather than vibes. It runs on CPU.

For the arm, the schema is tiny:

QUESTIONS = {
    "gesture": {
        "type": "choice",
        "instructions": "Which robot arm gesture does `command` ask for?",
        "criteria": {
            "wave": "wave hello or goodbye",
            "point": "point at something or a direction",
            "nod": "nod yes, agree",
            "shake_no": "shake no, disagree",
            "bow": "bow, thank, show respect",
            "grab": "grab, pick up, hold",
            "release": "release, drop, let go",
            "dance": "dance, celebrate",
            "rest": "stop, relax, rest position",
            "random": "do anything, surprise me",
            "none": "not an instruction to the robot arm",
        },
    },
    "speed": {
        "type": "score",
        "instructions": "How fast should the robot arm move for `command`?",
        "criteria": ["slow: gently, slowly", "normal: no speed mentioned", "fast: quickly, hurry"],
    },
}

Two things to notice.

The none option is what lets the arm listen all the time. "The meeting is at three o'clock" → none (99%) → nothing moves. There's no wake word because the model itself decides whether you were talking to it.

Speed is a second question on the same pass. "धीरे से झुको" (bow slowly) gives bow at 97% and a speed score of 0.0 (slow) from the same ~120 ms of compute. The motion engine turns that into a duration multiplier.

Hindi in, bow out, speed = slow. The arm is mid-bow: shoulder 140°, elbow 135°.

Stock Laya wasn't good enough for Hinglish

Out of the box, on my held-out test set, Laya's multilingual checkpoint got 43.7%. English was fine-ish (63.5%), but romanised Hinglish — "nach ke dikhao", "udhar dikhao" — was 31%. Not surprising: "haath hilao" isn't in anyone's pretraining data as a robot command.

So I fine-tuned it. Laya ships a training loop (laya.train.finetune) that takes JSONL rows of {state, questions, expected}.

The dataset is synthetic, and the split is the important part

I don't have a thousand recorded commands. I have phrase banks:

Each phrase gets wrapped in templates ("robot, …", "zara …", "कृपया …", "… please") and sometimes a speed word ("jaldi se …", "… slowly"). That gives 1,154 training rows and 135 test rows.

The part I'd argue is non-negotiable: whole phrasings and whole templates are held out. The test set contains wording the model never saw during training. If I'd split on rows, the same phrase would appear on both sides with a different wrapper, and I'd be measuring memorisation.

Training

Four epochs of soft cross-entropy on the multilingual checkpoint, with Laya's option-order shuffling turned on so the model can't learn "the third option is usually right". About an hour on a laptop GPU (Apple M-series), including the before/after evaluation. The result is a ~650 MB fp16 checkpoint that loads with laya.load("arm/model").

Results

Laya's own calibration report on its in-sample slice went from ECE 0.20 → 0.007, which is the number that makes the next section work.

The confidence gate

The arm only moves if Laya's confidence in the chosen gesture is at least 0.45. Because the confidences are calibrated, this threshold behaves the way you'd hope:

At the default gate it performed a wrong gesture on 12% of held-out commands, ignored 13 real commands, and reacted to 2 of 18 chatter sentences. Raise the gate and it gets more cautious; lower it and it gets more eager. One number, no retraining.

"the meeting is at three o'clock" → No action, 99%. The arm stays at rest.

Voice: two recognisers, one decision

Speech-to-text runs locally with Vosk. The small models are ~50–80 MB each and decode a two-second command in a few hundred milliseconds on CPU.

The problem is that no single small model handles both languages. The Indian-English model hears "wave hello to everyone" fine and turns "हाथ हिलाओ" into "hello". The Hindi model does the reverse: "नाच के दिखाओ" comes out right, in Devanagari, which Laya reads natively, while English becomes "लीज से नौ".

So both models decode every utterance, both transcripts go to Laya in one batch, and the more confident non-none decision wins. Laya, not a heuristic, decides which hearing to trust.

Spoken "नाच के दिखाओ". The English model heard "k dick how" (→ random, 60%); the Hindi model heard "नच के दिखाओ" (→ dance, 85%). Dance wins. Speech-to-text 561 ms, Laya 162 ms.

I use webrtcvad for endpointing so both recognisers see identical utterance boundaries, and a 600 ms silence closes an utterance.

Motion: keyframes, easing, and blending from wherever you are

The arm is a 4-DOF acrylic kit: base turn, shoulder lean, elbow lift, claw. Each gesture is a list of (duration_ms, partial pose) keyframes:

"wave": [
    (500, {"shoulder": 70, "elbow": 45, "claw": 0, "base": 90}),   # raise
    *repeat([(320, {"base": 62, "elbow": 38}), (320, {"base": 118, "elbow": 52})], 3),
    (350, {"base": 90}),
    (600, HOME),
],

A 50 Hz loop eases between keyframes with a cosine curve, so every joint starts and stops together; that's the difference between "deliberate" and "servo-jerky". Three details that mattered more than I expected:

• A new command blends from the current pose. Saying "stop" mid-wave doesn't snap the arm back to a start position first; the new timeline begins wherever the arm is.
• Velocity is capped per tick (360 °/s) on top of the easing, because an SG90 under load can't follow a step and will just buzz.
• Speed from Laya is a duration multiplier (0.6× for fast, 1.7× for slow), applied to the whole timeline.

Output goes to a PCA9685 PWM board over I2C (a 60-line driver using smbus2, no vendor SDK), or to the simulator you see in the screenshots, which draws the arm from the same joint angles every screen receives over a WebSocket.

Spoken "please say no" → SHAKE NO. Base swinging, 100%, 164 ms.

What it costs

About ₹1,500 (~$18) on top of the Pi. The arm uses ~2.3 GB of RAM (Laya runs in fp32 on the CPU), so an 8 GB Pi is the comfortable choice.

Where it fails

I'd rather you hear this from me than find it in the repo.

• The demo is the simulator. The servos aren't wired yet. The motion engine and driver are written and the Pi image is built and tested in emulation, but I have not watched real plastic move.
• No Raspberry Pi timings yet. The ~0.1 s per decision is a laptop CPU. In an emulated arm64 container it was 290–490 ms; a real Pi 5 will land somewhere, and I'll publish the number when I have it.
• Short Hindi words are hard for the small speech model. Spoken "रुको" (stop) was heard as "को"; the gate kept the arm still, which is the right failure, but it's still a failure.
• "haan bolo" (say yes) sometimes comes out as shake no. More varied training phrasings is the obvious next step.
• Fixed vocabulary. It can only do the gestures I defined. "Wiggle only the claw slowly" is not an option until I add it.
• No vision. "Pick up the red cube" needs a camera and that's a different project.

What I'd take away

The interesting part isn't the arm. It's that a decision model is a better interface between speech and actuators than a generative one for a bounded task: it's faster, it can't invent actions, and calibrated confidence gives you a safety knob that actually works. The fine-tune took an afternoon and a synthetic dataset. The same pattern would fit a smart light, a wheelchair control, a kiosk — anything with a fixed menu of actions and users who don't speak textbook English.

Next up: wiring the servos, measuring on the real Pi 5, and a five-finger hand so the gesture list can grow.

Repo: github.com/dockndevai/pi-meeting-assistant · Laya: github.com/NandhaKishorM/laya · Vosk: alphacephei.com/vosk

If you build one, or break it, I'd like to hear about it.