I get weird looks when I tell people my AI home lab is a folder. Not a rack, not a fancy app, a folder called ~/lab with a handful of docker-compose files in it.
That wasn't the plan. I started the way everyone starts: installed Open WebUI to try it, installed LM Studio to switch models, then a translator here, a little whisper model there. A month in I had five things running and no idea which one owned which port. Restarting the machine was a two-hour archaeology project.
So I got annoyed and Dockerized the whole thing. Now the lab is reproducible, and that turned out to be worth more than the AI itself.
Why compose, specifically
Anything in docker compose up -d can be torn down and rebuilt. That's the whole pitch. When a broken model update takes Open WebUI down, I don't debug for an hour. I delete the container, compose up again, and it's back.
The volume mounts are the real story. Keep your data on the host and the containers become disposable. Wipe a container that got poisoned by some experiment, your chats and settings survive in a mounted folder. The lab isn't precious. That's the point.
Size hardware to the models, not the marketing
Building this fixed my other mistake too: I almost bought a monster GPU because the internet told me you need one for "local AI." You don't. I ran the models I actually use on my existing machine and watched what fit. Most of what I run daily is an 8B model quantized down to 4 bits, and it fits in a few gigabytes of VRAM. My old laptop with no GPU at all runs it acceptably on CPU if I'm not in a hurry.
Write your real workloads down before you spend money, not the other way around. Figure out the three models you'd actually run daily, check their quantized sizes, then buy the hardware that fits them. I skipped a few hundred dollars of GPU because I did the math first.
The best part is the boring parts
What sold me wasn't any single model. It's knowing ~/lab and nothing else is the lab. Disk runs low, I find the compose file with the biggest volume and prune. Box dies, I rebuild an afternoon's work and pull the images. There's no cloud account to cancel, no monthly rental to justify, and no remote service that knows my chat history.
That localness is the whole reason I stick with it. My prompts, my model weights, my data, all on one box I control. The GPU question and the privacy question both got easier the day I stopped treating this like a product and started treating it like infrastructure.
If that setup sounds appealing and you'd rather start from a working bundle than assemble it piece by piece, I put a home-lab kit together that gets Open WebUI, quantized models, and the compose-based layout running fast. It's pay-what-you-want at https://symshah.gumroad.com/l/selfhosted-ai-homelab if you want to skip the archaeology.
