I Built a Local AI Music Studio

I have a habit of starting projects because I want one very specific thing.

Then somewhere along the way I apparently decide, "Well, if I'm already doing this..."

And suddenly the tiny experiment has a database, Docker Compose stack, job queue, model manager, audio editor, waveform player, and enough GPU models to eat 40 GB of disk space.

So, anyway.

I made Miso.

Miso is a self-hosted local music generation and remix studio built on top of audio.cpp.

The original idea was pretty simple: I wanted a nice interface for experimenting with local music models without constantly dropping back to command-line tools, juggling generated files, or trying to remember which ridiculous combination of parameters made the song I actually liked.

It got slightly out of hand.

I didn't just want "type prompt, get song"

There are already plenty of ways to generate a song from a text box.

That part is fun, but the thing I kept wanting was everything that comes after generation.

What if I like the song except for eight seconds in the middle?

What if I want the vocals by themselves?

What if I want to change the singer?

What if I want the same melody arranged in a completely different style?

What if I want to turn something into MIDI and keep screwing with it somewhere else?

That became the real idea behind Miso.

A project doesn't just hold the final song. It keeps the takes, prompts, lyrics, stems, scores, transcriptions, and the lineage between them, so I can experiment without losing track of how I got there.

There are a ridiculous number of models involved

Miso currently exposes several different music and audio model families through audio.cpp because they are good at very different things.

ACE-Step 1.5 is the main workhorse. It handles full song generation, covers, and repainting sections of existing tracks.

YuE2 can generate songs too, but it does something I find especially interesting: it plans an ABC music score first and can give that score back to you.

MiniMax Music 3 and HeartMuLa are alternate song generators with their own prompting styles.

Stable Audio 3 handles instrumental generation and sound effects.

Then there are separate models for stem separation, voice conversion, and audio-to-MIDI transcription.

The funny part is that they absolutely do not accept prompts the same way.

One model wants a production-style caption.

Another wants a short description plus comma-separated tags.

ACE-Step wants style information in one place, BPM and key as separate parameters, and lyrics somewhere else.

So Miso has a guided builder that translates the same set of UI controls into whatever syntax the selected model actually expects.

I wanted to be able to think about "upbeat synthpop, female vocal, 120 BPM" instead of remembering which API field each model decided should contain "120 BPM."

Repainting is interesting

One of the features Miso is most built around is repainting.

You load a take, drag across part of its waveform, and replace only that section.

So instead of regenerating a whole song because one chorus went weird, you can select that section and try again.

ACE-Step's repaint route turned out to have some quirks that were interesting to figure out, too.

The surrounding music influences the replacement much more strongly than the text prompt does. Lyrics have more control over a vocal repaint than telling the prompt to suddenly turn the section into something wildly different.

That wasn't something I wanted to guess about, so I ended up testing it.

At one point I ran deliberately opposite prompts through the route and compared the results. The normal text-to-music route reacted dramatically. Repaint barely cared.

Which is one of the reasons I enjoy projects like this. Half of building the interface becomes figuring out what these models actually do rather than what I assume they do from an API description.

Then I started pulling songs apart

Miso can also split a track into stems.

Depending on the model, that can mean:

• vocals + instrumental
• vocals + drums + bass + everything else

Once the stems exist, they get their own mixer.

You can solo them, mute them, change their levels, swap a vocal using a voice conversion model, mix everything back together, and save the result as another take.

That led to another rabbit hole: voice conversion.

RVC sounds better but only has a small set of packaged voices.

Vevo2 has lower output bandwidth, but it can use basically any other vocal reference in the project.

So both stayed.

I would rather expose two tools with different tradeoffs than pretend one is objectively the "correct" choice.

The part I just finished is more interesting

The current release, 1.4.0, connects MIDI transcription to YuE2's score support.

Previously, Miso could already:

1. split a song into stems
2. transcribe audio into MIDI notes
3. give YuE2 an ABC score and tell it to sing that melody

The pieces just weren't connected.

Now they are.

You can separate a song, transcribe the vocal stem, turn those note events into an ABC melody, inspect the detected tempo and key, correct them if necessary, and feed the score into YuE2 with a completely different style.

So you can take the melody from a recording and use it as the basis for a new arrangement.

I tested that path with a known melody instead of just listening and declaring, "yeah, sounds about right."

I generated a simple version of Twinkle Twinkle Little Star, ran it back through separation, transcription, and score conversion, and recovered the expected pitches across the recording.

Rhythm is less perfect than pitch, and real vocals can produce odd little tied or split notes, but the result is usable by the model.

That was one of those features where suddenly several earlier pieces of the project made much more sense together.

Miso is local-first

The whole thing runs as two Docker containers.

One runs Miso itself.

The other runs audio.cpp and owns the GPU models.

The browser never talks directly to audio.cpp. It talks to Miso, and Miso talks to the inference backend.

That separation became useful almost immediately.

Projects, imports, playback, exports, and editing still work if the model backend is stopped.

It also means the audio.cpp server can live on another machine with the GPU while Miso runs somewhere else.

The app itself uses React, TypeScript, Vite, Node, Hono, and SQLite.

I deliberately kept most ordinary audio editing in the browser rather than turning everything into another server job.

Things like trimming, fading, gain changes, normalization, splitting, WAV conversion, and MP3 export do not need a giant AI model involved.

Sometimes software can just... do the normal thing.

Wild concept, I know.

There's an optional AI assistant too

I did sneak one language model feature in.

Miso can optionally connect to an OpenAI-compatible API or a local llama.cpp server to help write lyrics or expand a music prompt.

But it is completely separate from the music models.

Nothing gets silently rewritten.

It gives you a suggestion, shows it to you, and you decide whether to use it.

That was important to me because I don't want an assistant "improving" a prompt behind my back and then making it impossible to figure out why two generations were different.

If you accept an expanded prompt, Miso keeps both the original and the version that actually ran.

This project taught me a lot about building around AI models

Not really about calling them.

That part is usually the easy bit.

The interesting problems were things like:

• models that require different sample rates
• models that return multiple huge audio files
• models that need several components loaded in sequence to fit in VRAM
• jobs that can run for several minutes
• uploads that need to survive browser tabs closing
• model APIs that technically accept a parameter but quietly ignore it if you put it in the wrong part of the request
• keeping enough generation history to reproduce something useful
• dealing with a backend that can only run one expensive job at a time
• determining which behavior belongs in the UI versus the server versus the inference runtime

And, of course, learning that "the documentation says this option does X" and "the model meaningfully responds to X" are not always the same statement.

That last one is probably applicable to a lot more than music models.

It's still very much a project

Miso is open source, and right now it has mostly been tested on my own Linux machine with an RTX 4090 laptop GPU.

So I am absolutely not going to pretend I have tested every GPU, driver, operating system, or model combination in existence.

But it has grown from "I want a good way to play around with local music models" into something I actually think is pretty awesome.

And that is my favorite kind of project.

The kind where I started building a tool because I wanted to experiment with something...

...and then ended up building the experiment itself.