This is a submission for the MLH x DEV Writing Challenge
Built at DubHacks 2026 with @maiaPaperTowns, @alesia-cameron, @matthewcflam and me @lkim0402. ReMind supports engagement. It is not a diagnostic or treatment tool and hasn't been clinically tested.
What We Built
Our project is a suite of voice-driven memory game for older adults with dementia or memory loss, built for use in older-adult centers. The patient just talks, so there are no menus, and the screen stays calm, showing only one photo or a few words at a time.
- A caregiver stays in charge of each session, and manages patient records through a patient dashboard.
- A suite of memory games including a shuffle game, with a real robot arm so the experience isn't only on a screen.
The problem and the research
Cognitive stimulation, meaning enjoyable group activities that exercise thinking and memory, has a research base: a 2023 Cochrane review (Woods et al.) found small, short-term cognitive benefits for people with mild-to-moderate dementia. Digital tools exist, but reviews keep naming the same four gaps. They aren't personal enough, since personalization is a recurring challenge and most exergaming systems lack features for personalized care. They're hard for older adults to use, because existing exergames mostly target younger people (2026 usability study). People stop using them: up to half of facility-based sessions are missed (2026 AAN trial abstract) and adherence to home programs declines over time. And the evidence is thin, with few, small trials that vary widely in duration, frequency and platform. We designed against all four with memory-jogging games based on cognitive science, voice-only interaction, a calm one-photo screen, caregiver control, and a robot arm that keeps sessions engaging.
Tools we used
- Frontend: Next.js 15, React 19, TypeScript
- Backend: Python, with MongoDB for data
- Voice and AI: Gemini Live for real-time voice conversation, Gemini for chat, and ElevenLabs for speech
- Memory: Backboard
- Messaging: a Photon service
- Robot: SO-101 arm run by a separate robot worker, with side and wrist cameras and Perceptron for vision
Demo
How it works
There are four parts: a voice companion that listens and replies in real time in short sentences, a calm screen, a robot arm with a camera, and the caregiver, who sets up each session, reads the summary afterwards and gets a text if something seems off.
Partner Technologies
The main game: The Shuffle Game (SO-101 arm, trained on Perceptron)
Why an arm? Most digital activities sit behind a screen. The arm lets the resident do something with their hands, with real objects, and the activities stay really simple, like putting three things in order. We also see it as a starting point. We want to teach it more policies (new skills) over time, and each one could become a new hands-on game.
- The arm mixes up three keychains on a table.
- The resident puts them back in the right order.
- A camera reads the order they just made. The check returns a plain result: right or wrong, why, and how many tries are left.
- The AI gets that result and answers in plain words, like "Almost! Try swapping the last two." The resident gets three tries. After that, ReMind shows the answer gently. There's no score and no timer.
- A one-line note about how they did goes to the caregiver. It's an observation, not a grade.
The camera read is live, and it's the part we like most. The AI never decides on its own whether the order is right. It gets a result from what the camera sees and turns it into something kind.
Teaching the arm
We wanted the arm to shuffle three keychains (black, white and purple) five ways: swap left and right, swap left and middle, swap middle and right, rotate everything one spot left, and rotate one spot right.
We didn't program the robot's movements, rather the SO-101 arm learns from examples, a bit like being taught a card trick by demonstration: one person moves a "leader" arm by hand while a second "follower" arm copies every move. Two cameras film each try, one watching the table from the side and one on the gripper. We recorded 60 tries of 35 seconds each, 12 for each shuffle, and uploaded them to Perceptron's cloud platform to train. Then we test it again and again on the real table and count how often it works. When it fails in a specific spot, like a keychain sitting a bit too far left, we record more examples of that case and train again.
A few things we learned:
- The arm is picky about its surroundings. Table color, lighting and camera angle all have to match between recording and use, so we taped everything down.
- It copies how you move, speed included. We demonstrated slowly and gently, since it works near older adults.
- A bad try teaches a bad habit, so we redid those. We also nudged the keychains a little each time. Without that, the arm gets confused the moment something is slightly off.
The other games: voice-only, powered by the Gemini Live API
The arm is one game out of 4 included games. The rest run on voice alone (built on Gemini's Live API), which listens and talks at the same time and lets you interrupt. They follow session themes from Cognitive Stimulation Therapy:
- Famous faces: a photo appears, ReMind asks what they notice and what it reminds them of, then reveals the name.
- Sayings: ReMind starts a well-known saying ("A stitch in time saves...") and the resident finishes it.
- Sounds: ReMind plays an everyday sound, then asks what it was and what it brings to mind.
On the technical side, the browser streams the resident's voice to our backend, which relays it to Gemini so the API key stays off the browser. Gemini can call a short list of tools (start_game, reveal, next_item, play_sound, show_message, note_response, end_game, stop, alert_caregiver), and the screen shows only what those tools put there. The prompt asks it to ask for opinions and memories instead of facts, and never to test or correct anyone.
Texting the caregiver: Photon SDK
If a resident says something worrying, we want the caregiver to know without scaring the resident. Gemini decides whether something is worrying (pain, a fall, fear, feeling lost). If it is, it calls an alert_caregiver tool with a short, plain summary and no diagnosis, and calmly reassures the resident. The nurse enters a caregiver's phone number next to the patient's name, and the text reads like: "ReMind alert, 3:53 AM, Margaret: Said her chest hurts and she feels dizzy."
It sends in the background, so a slow or failed text never interrupts the conversation. We send it through Photon as an iMessage. Its SDK is TypeScript and our backend is Python, so a small Node service bridges the two. The weak spot is Gemini's judgment: it might skip something borderline or fire on something mild, and nothing yet limits how many texts one worry can send.
Long-term memory (powered by Backboard)
Each resident gets their own Backboard assistant, named after their participant ID, so one person's memories never mix with another's and survive even if our own database is wiped. It holds what grows over time: people in their life, favourites, stories they like to tell, topics that upset them, and what engaged them. Their basic care profile stays in our own store.
During a call, Backboard Realtime routes the voice session to Gemini Live and retrieves memories read-only, because saving facts mid-conversation delayed replies by 1 to 2 seconds (once 10) and stored every retelling of a story again. Instead, after the call we extract only the new facts from what the person said, drop any already remembered, and save the rest in the background, with no medications, diagnoses or test scores ever recorded. Nurses can view, edit and delete what the companion remembers, and memory only runs with consent, which can be turned off to delete everything. If Backboard is slow or down, a session never fails or waits on it.
Hackathon Experience
We participated in DubHacks 2026, and it very memorable and fun, we learned a lot! Brainstorming took a lot of the day: we spent until around 5 pm just deciding what to build. When we settled down on ReMind, none of us had any robotics experience, so building around a real robot arm felt like a leap of faith.
Getting it to work was the hardest part, and when we got it working at around 4 am after hours of trying to get the robot arm move and get it trained, it felt extremely rewarding!
The Perceptron team were extremely willing to help and guided us throughout, and we're really grateful they were there for us.
In the end we won 2nd place in the Perceptron Robotics Track and the MLH Backboard Prize!
What's next
- More skills for the arm. Each new policy we teach it could become a new hands-on game.
- A caregiver dashboard that shows plain facts for a whole room of residents, not a decline score.
- Games that adapt to each person, which is how we'd properly answer the personalization gap. Today we only pick games per person and use their name.
- Consent and data controls, and a pilot with a care facility and a clinician before we claim anything.
