This is a submission for the MLH x DEV Writing Challenge
What We Built
Roblotix is a set of Roblox games played by two people at once: a player at the Roblox screen and a robot operator at a real SO-101 robot arm. The blocks the operator moves on a real table become platforms, bridges and targets in the game. The game can also reach back and shake, slow or flip the real arm.
We built three games on one shared core (roles, a game registry and a bridge to the arm):
- Robo-Obby: Smart Warehouse Escape. The player is stuck in a warehouse of hazards they can't clear alone, and the operator builds the way through with real cubes: a closing saw wall, a gap in the floor, a laser floor and a dark room.
- Pattern Match. A ticket-shift game. The player sees a target pattern the operator can't see, and calls out moves by voice while the operator rebuilds it on the real grid.
- Dragon Escape. ** The robot arm controls a dragon with fire breath. Players have to survive the timer and can use powerups to slow down the robot arm in real life.
The idea came from a realization: robot arms and roblox have never met. We wanted the arm to be a teammate inside the game, not a controller for it.
The detail I'm proudest of is one number. With the mat scale $s = \frac{8\ \text{studs}}{57.15\ \text{mm}}$ and a $48\ \text{mm}$ cube, one real cube becomes
$$48 \times s \approx 6.72\ \text{studs}$$
A player can climb one cube but not a stack of two ($13.44$ studs), so the physical world sets the difficulty. If the operator doesn't stack cubes, the lever gate never opens.
What We learned:
- Keep the physical and the virtual on one coordinate system. It's easier to build around real constraints than to patch the game later.
- Design for noisy input. Hold times, debouncing and forgiving tolerances matter more than clever vision.
- Ask a vision model only when the answer can have changed. Our first approach tracked blocks frame by frame, which took hundreds of lines and broke whenever the arm carried or hid a block. We replaced it with one vision read two seconds after the gripper opens.
- Build a fake version of every dependency. A mock bridge and a solo mode let three people work at once without the hardware.
Demo
[https://youtu.be/l9G8ekAVupE]
Code: https://github.com/abhay-dronavalli/LeBot-Games
Partner Technologies
Google Gemini API. Gemini names block colours in our perception pipeline. We send the overhead frame with each detected cube outlined and numbered, and ask for a JSON reply mapping each number to a colour name. We use it to teach our colour recognizer at the start of a session or when a block looks unfamiliar, so everyday recognition stays local and fast.
What worked well:
- The prompt fits in a few lines, and it followed "ignore glare, shadows and printed patterns" better than the classical colour thresholds we started with.
- Asking for JSON only gave us replies we could parse without retries.
- Choosing the newest "flash" model automatically saved us from model-name churn mid-weekend.
What I'd improve: a clearer quota signal. Our rig looked healthy while blocks stopped updating after the credits ran out, so we added a visible warning banner.
Hackathon Experience
I built this at SwampHacks XII at the University of Florida. It was me (Kushagra Katiyar) Abhay Dronavali, Taher Akalawala. It was very fun I drove to walmart to get children's toys and my carts never seemed that way
in the last 6 years.
Hardware hacks are different from software ones. A software bug waits for you; a servo bus that drops out 40 seconds into motion does not. The part I'll remember is when we saw roblox reflect the movement of the robot for the first time. It felt surreal two people playing the game, one physicall and the other in the game.
