My journey into exploring on-device artificial intelligence recently led me to an innovative project: a 125M-parameter model trained to autocomplete piano performances in real time. This work, detailed by SimEdw, showcases how thoughtful engineering can make powerful AI accessible directly on mobile devices, like an iPhone 15. It’s an exciting step towards bringing advanced AI capabilities into our everyday tools.
The foundational challenge for any language model, including those for music, is translating raw data into a sequence of discrete tokens. For MIDI music, this means transforming events like key presses, releases, and pedal changes into a format a transformer can learn from and predict. What started as an idea to create a 'GitHub Copilot for piano' quickly turned into a deeper exploration of music tokenization strategies.
Architecting for On-Device Efficiency
Early attempts at MIDI representation highlighted critical trade-offs between expressive power and computational efficiency.
- Event-based tokenization: Mapping every MIDI event (pitch, velocity, note-on, note-off) created a very large vocabulary. Models using this struggled with "drift," often forgetting to emit note-off events or losing track of active notes. This made real-time inference difficult, especially for small models.
- Grammar-enforced sequences: A slightly more structured approach involved tokens like
[NOTE_ON, PITCH, VELOCITY]or[NOTE, PITCH, VELOCITY, DURATION]. While solving some drift issues, these often required multiple autoregressive transformer steps per musical note, significantly slowing down generation. One such representation made a single musical note require roughly four transformer passes, burning through the context window rapidly.
The breakthrough for this project came with a novel approach: representing an entire musical note as a single, composite token.
Here's what made the final representation so effective:
- Single Note Token: Instead of breaking down a note into separate events, the model processes
NOTE(pitch, delta_onset, duration, velocity)as one unit. This means the transformer advances the music by one complete note at a time, not by individual attributes. - Categorical Fields and Embeddings: Each note internally consists of five categorical fields:
event_type,pitch_id,delta_id,duration_id, andvelocity_id. Each field receives its own embedding, and the final note token is the sum of these embeddings. - Separate Output Heads and Nested Decoder: The model predicts each field using separate output heads. A small nested decoder then conditions later fields on the earlier predicted fields. This allows for intricate note generation without the full computational cost of running the main transformer backbone multiple times per note.
- Sustain Pedal Integration: To simplify the modeling problem, sustain pedal events are not explicitly represented. Instead, the effect of the sustain pedal is baked directly into the note duration during preprocessing, approximating the actual sounding duration. This means the model only needs to predict pitch, onset, duration, and velocity.
In simple terms, by intelligently compressing the musical information into a single token per note and optimizing the internal prediction process, the model can generate music much faster and more reliably. It achieves a remarkable speed of about 108 notes per second on an iPhone. This speed is well beyond what a human player would need for live interaction.
What started as an idea to bring AI autocomplete to piano players turned into a hands-on exploration of efficient model architectures and data representations. This journey reveals that pushing the boundaries of AI often means a deep dive into the fundamentals of how we structure information for our models. It underscores the importance of adaptability in our engineering approach.
This project is a powerful example of how focusing on engineering for efficiency and real-world deployment can unlock new possibilities for agentic AI systems. It reminds us that our role as builders is not just to consume AI, but to contribute to its practical and responsible application.
