Everyone who's ever pitched a learning app has said the same sentence: "It's like Duolingo, but for X."
We said it too. Ours was singing.
Here's the thing nobody tells you: Duolingo works because language has discrete, checkable answers. "Der Hund" is either right or wrong. A green owl can grade that in one if statement.
Singing has no if statement. You can hit the right note with terrible breath support, strained vocal cords, and a jaw so tense you'll be hoarse by Thursday. A naive app will give you a gold star for all of it — and quietly teach you habits a real vocal coach would spend months undoing.
This is the story of how a small team (three engineers, one designer, one very patient vocal coach as an advisor) built Singing Coach AI, what we got embarrassingly wrong, and why we ended up betting everything on multimodal AI analysis instead of gamification.
Lesson 1: Pitch detection is table stakes, not a product
Our v0 was what you'd expect: a pitch detector (autocorrelation, later CREPE-style neural pitch tracking), a piano roll, and a score. Sing the note, watch the line, get a percentage.
It demoed great. It retained terribly.
Users would nail the pitch-matching exercises within a week and then... plateau. Hard. Because pitch accuracy is maybe 20% of what makes someone a good singer. The other 80% — tone quality, breath control, vowel shaping, tension, dynamics — was completely invisible to our system.
We were the equivalent of a language app that only checks spelling. Congratulations, you spelled "der Hund" correctly. Your pronunciation? No idea. Your grammar? Not our department.
Takeaway for anyone building AI learning tools: if your model can only see one dimension of the skill, your feedback will optimize users toward that dimension at the expense of everything else. Goodhart's law, but for hobbies.
Lesson 2: Streaks don't fix bad feedback
Naturally, when retention dipped, we did the startup thing: gamification. Streaks, XP, leagues, a mascot (RIP Melody the songbird, 2024–2024).
It moved the metrics for about three weeks. Then we read the user interviews and felt sick:
"I keep my streak going but I don't think I'm actually getting better."
Duolingo's gamification works on top of a feedback loop that's genuinely correct. Ours was gamification on top of feedback that was 80% blind. We were paying people in dopamine to run on a treadmill.
We killed the leagues and went back to the actual problem: what does a human vocal coach perceive that our system didn't?
Lesson 3: A vocal coach is a multimodal system
We sat in on real lessons with our advisor and took notes on what she actually reacted to. It was humbling:
- She heard pitch, timbre, breathiness, vibrato onset, and where the resonance was sitting.
- She watched posture, jaw tension, shoulder movement on inhales, and mouth shape on vowels.
- She tracked context over time — "your break is smoother than last month, but you're pushing chest voice higher than you should."
No single-signal model gets you there. So we rebuilt Singing Coach AI around multimodal analysis:
Audio, but deeper than pitch. Beyond f0 tracking, we analyze spectral features for tone quality and breathiness, formants for vowel placement, onset characteristics, and vibrato rate/extent. Pitch says what note you sang. Spectral analysis starts to say how — and "how" is where vocal technique lives.
Video for technique. With the camera on (optional — audio-only mode exists), pose and face landmark tracking flags the stuff singers can't feel themselves doing: jaw clench, raised shoulders on breath, collapsed posture at phrase ends. Fun engineering note: shoulder-rise-during-inhale turned out to be one of the strongest single predictors that a user would report vocal fatigue later. A pitch-only app would never know.
LLM-driven coaching on top. Raw signal analysis produces numbers; numbers aren't coaching. The layer users actually interact with takes the multimodal features plus their history and generates feedback the way a coach would phrase it: "You're going flat on sustained notes at the ends of phrases — that's usually breath support running out, not a pitch problem. Try this exercise." Same data, radically different usefulness.
That last part mattered more than we expected. Telling a beginner "cents deviation: -34 on measure 12" is useless. Telling them why it's happening and what to do about it is the product.
Lesson 4: Small team, so cheat wherever the domain allows
Things that saved us as a five-person team:
- Don't train foundation models. Compose them. Pretrained pitch trackers, pose estimation, and speech/audio embeddings got us 80% there. Our proprietary work went into the fusion layer and the vocal-pedagogy-specific fine-tuning — the part nobody else had.
- On-device first for the hot path. Real-time pitch feedback has to be <50ms or it feels broken. That runs client-side. The heavier multimodal analysis runs async after the take — users happily wait 5 seconds for a "coach's review" they'd never tolerate as live latency.
- Your domain expert is your eval set. Our biggest quality unlock wasn't more data — it was having our vocal coach grade the AI's feedback on ~500 real takes and hammering the disagreements. Cheap, brutal, effective.
- Curriculum > content volume. Duolingo has thousands of lessons. We launched with a fraction of that but sequenced by an actual pedagogy (breath → pitch → resonance → repertoire). Depth of feedback made the thinner catalog feel richer, not poorer.
What "Duolingo for singing" actually means (now that we've built one)
It's not the owl. It's not streaks. It's the property that made Duolingo work in the first place: a tight loop of attempt → accurate feedback → targeted next exercise, available at 11pm in your bedroom for a fraction of the cost of weekly lessons.
Human vocal coaches are wonderful and irreplaceable for advanced work — and $60–150/hour, once a week, if you're lucky enough to live near a good one. The gap we're filling is everything in between: daily practice with feedback that actually sees (and hears) what you're doing, not just whether you hit the note.
That's the bet behind Singing Coach AI: multimodal AI analysis — audio, video, and longitudinal progress together — is the difference between a pitch game and an actual coach.
If you're building "Duolingo for X"
Ask yourself one question before writing a line of code: can my system perceive the skill the way an expert does?
If the answer is no, gamification will only help you retain users while they build bad habits faster. Fix the perception problem first. Everything else — streaks, XP, even Melody the songbird — is optional.
We're a small team and we ship fast. If you sing (or gave up on singing because feedback was impossible to get), try singingcoach.ai and tell us what the AI got wrong — those reports are literally our roadmap.
Questions about the pitch tracking, the pose pipeline, or the LLM feedback layer? Ask in the comments — happy to go deep on any of it.
