Being a robotics and vision AI grad student at NAIST, I spend my days with a real robot arm, cameras, and a long list of failed experiments. Here's what I've learned about where things actually break.
The boring problem
People assume the hard part is the model, but for us, it's writing the Python that glues perception to motion, and every time we swapped models, we ended up rewiring the same motor functions. What changed our workflow was treating robot capabilities as reusable skill bundles (Skillware): write the deterministic script once, and any model can call it. You swap the brain without rewiring the motor functions. The problem is Skillware is mostly software focused, and although it gives you a good baseline to create your own skills, it still takes time to write scripts the first time, but then it works like magic.
Determinism is a feature, not a limitation
A robot has a fixed set of skills, and it cannot invent new ones mid-task, which is exactly what you want, that's why a deterministic skill either works or it doesn't, fits robots the best. No grey zone, no hallucinated torque values, and honestly, half of our bugs aren't AI bugs at all, but unit mismatches, degrees fed into radian-expecting nodes, mg instead of m/s².
The tradeoff is context I guess, when 20+ skills loaded in full, the context gets cluttered fast. So we only append short descriptions of what each skill does and when to use it, and the full context unlocks only when a skill is triggered. It basically doubles as smart memory management.
Knowing why the robot did something
On top of the skills, we run a harness (aura) with a native sequencer, which gives us a real audit trail, like what the robot did, when, why, and where the reasoning came from, was it the model's own thought, something from a skill bundle, or something picked up from an internet post? With physical hardware, "it kind of seemed like a good idea" is not an acceptable answer.
The real world refuses to cooperate
In vision, the hardest part is lighting and unpredictable background noise, people walking by, hands entering the scene, and here's the ironic one: even when the robot does everything perfectly, strict guardrails can make it freeze. It sees a hand in the frame, can't compute distance reliably from a frozen snapshot, decides that moving might risk contact, and freezes rather than move, which yes, is perfectly safe, but also completely useless. The robot isn't wrong, but our guardrails just have no concept of "re-plan from a new viewpoint."
So the model is maybe 20% of the problem. The rest is deterministic skills, context management, audit trails, and a world that refuses to hold still. If you're building agent pipelines for hardware, invest there first. What keeps breaking in your robot stack? Happy to trade war stories :D
