Building a 512-LED wall that dances to my music: FFT, ESP32, and a custom serial protocol

Context

A while ago I decided my wall needed to react to music. The build is 512 WS2812B LEDs — eight 8×8 panels arranged as a 32×16 grid — driven by an ESP32. The design decision that shaped everything came early: the PC does all the analysis, and the ESP32 is a deliberately dumb display. The PC decodes the mp3, FFTs it into 32 frequency bands, builds a 16×32 RGB frame, packs it, and streams it over USB serial at 921600 baud. The ESP32 receives 1542-byte frames, applies a 25 A current limiter, and hands them to FastLED. The interesting engineering is in that split, the protocol between the two halves, and what I had to debug to make it not corrupt.

Approach

The architecture: PC is the brain, ESP32 is the wall. The PC already owns the audio stream, so doing the analysis there means zero duplicated state. soundfile decodes the mp3, numpy computes the FFT into 32 bands, a vectorized HSV→RGB frame builder renders a 16×32 image, and pack.py serializes it. The ESP32 firmware only parses frames, runs the current limiter, and calls FastLED.show(). Palette and geometry changes are Python edits — no firmware re-flash needed to tune the visuals. That split is why the whole project lives comfortably in two small codebases instead of one hairy one.

The protocol had to be engineered, not improvised. Each frame is exactly 1542 bytes: 2-byte magic (0xAA 0xAA) + 2-byte length + 2-byte sequence number + 1536 raw GRB bytes (512 LEDs × 3). The numbers on the wire dictated everything else:

For context, 921600 baud is roughly 90 KB/s — nothing exotic by USB standards, but at that rate a 1542-byte frame takes 16.7 ms on the wire, which is the same order of magnitude as the render time. That's why the rate math matters: the three fps numbers (24 shipped / 27 measured under lock-step / 31 hard ceiling) sit in a deliberate ladder — I ship conservatively below the measured ceiling so the wall never brushes the corruption edge in normal use.

The first version was fire-and-forget: the PC streamed frames and hoped. It didn't hold up — during FastLED.show() the ESP32 isn't reading the UART, and frames piled up and corrupted. The fix was lock-step flow control: the ESP32 sends an ACK byte (0x01) after every rendered frame, and the Python side holds the next frame until it arrives. Zero frames lost during LED update; a missed ACK degrades that one cycle to fire-and-forget instead of stalling the wall.

The rewrite that made it sample-synced. The original stack was pygame/tkinter. I replaced it wholesale with sounddevice + numpy: playback position now comes from the audio callback counter, so the FFT is sample-synced by construction — there's no drift between what you hear and what the wall shows, ever. Every tunable lives in one settings.py. The frame builder is numpy-vectorized and locked by a golden reference test.

Architecture

graph LR
    A[MP3 folder] --> B[soundfile decode<br/>libsndfile]
    B --> C[numpy FFT<br/>32 bands]
    C --> D[vectorized HSV→RGB<br/>16×32 frame]
    D --> E[pack: GRB + panel remap]
    E --> F[USB serial @ 921600<br/>1542 B/frame @ 25 fps]
    F --> G[ESP32 UART RX<br/>4096 B buffer]
    G --> H[magic / LEN / SEQ parse]
    H --> I[25 A current limiter<br/>protects 30 A fuse]
    I --> J[FastLED.show()<br/>15.86 ms floor]
    J --> K[8× WS2812B boards<br/>512 LEDs]
    J -->|ACK 0x01| F

Evidence

The wall playing "Slow Dancing in a Burning Room" by John Mayer — the 32×16 grid reacting to the vocal line and the guitar:

▶ Watch the live demo

The system was accepted on hardware in two tagged states: v1-working (the pygame/tkinter stack, preserved as a revert point) and v2-headless (the rewrite). The test suite is real: 8/8 test_pack.py, 2/2 test_render.py, 3/3 test_audio.py, plus a passing end-to-end pipeline_test. The frame builder has a golden reference test — the packed bytes are asserted against a precomputed output, so a geometry change can't silently break the wire format.

The git history reads as a debugging log, which is what it is:

b192b92 feat: lock-step flow control — ESP32 ACKs each render, zero frame loss
f2fbb6d perf: I2S DMA LED driver, numpy pack_frame, monitor_speed fix
0f20384 feat: acceptance hardening — seq=0 heartbeat resync, fps docs, diag off
0ef09f0 feat: 24fps acceptance — truthful counters, single pacing governor
8969187 feat: headless entry point (folder loop, synced FFT->serial)
70dd905 feat: numpy-vectorized frame builder with golden reference test

What went wrong

The 256-byte buffer that couldn't hold a 1542-byte frame. The default ESP32 UART FIFO is 256 bytes. Frames are 1542. While FastLED.show() runs (15.86 ms of not-reading-the-UART at 921600 baud), the FIFO overflows and the stream corrupts. The fix is one line — Serial.setRxBufferSize(4096) before begin() — but it took real staring at garbage frames to find. The lesson: your buffer size is a protocol parameter, and the default will betray you exactly when the hardware is busy.

I lost 50–70% of sent frames and shipped it anyway. Under lock-step, the ESP32 is busy in show() when frames arrive, so a large share of what the PC sends is dropped. This is now a characterized behavior, not a mystery: each delivered frame is still audio-synced, so the wall shows a slightly sparser but correct animation. I documented it as a known characteristic and accepted it rather than pretending I'd fixed it.

The first stack was throwaway. pygame/tkinter worked but made the timing story fragile. The rewrite to sounddevice+numpy was the difference between "roughly synced" and "synced by construction." I should have started there.

Known limitations

• ~50–70% of sent frames are lost to show() RX-starvation windows; every delivered frame is still sample-accurate.
• Hard ceiling ~31 fps because receive and render are serialized; shipped at 24 fps.
• The UDP path is dormant — serial is the only live transport.
• The 25 A limiter caps aggregate brightness: it scales all channels if Σ((R+G+B)/255)×20 mA exceeds 25 A, protecting a 30 A fuse at 30.7 A theoretical max draw.

Lessons learned

• A dumb-device/smart-host split keeps a hardware project small: re-tune visuals in Python, never re-flash.
• Flow control beats fire-and-forget at 921600 baud. An ACK byte is cheaper than debugging corruption.
• Measure the hardware floor before designing the protocol. 15.86 ms of show() time made the frame ceiling a math problem, not a guess.
• Buffer size is a protocol decision, and the default is a trap.
• Tests work on hardware projects too — the golden reference test means the wire format can't silently drift.

Links

• Repo: Shaarkymoo/music-visualizer — firmware (src/main.cpp), PC app (python/), full docs + parts list
• FastLED — the LED driver library
• sounddevice / soundfile — the audio engine
• WS2812B datasheet — the LED in question