My RTX 3060 Can Run Many LLMs — Just Not at the Same Time

Running a local LLM is easy.

Running multiple specialized models for coding, reasoning and AI agents on a 12GB GPU is where things get interesting.

My local AI machine is fairly ordinary:

• NVIDIA RTX 3060 — 12GB VRAM
• AMD Ryzen 7 5800X
• 64GB RAM
• Ubuntu Linux

Modern 7B–12B models run surprisingly well on this hardware.

But my workflow doesn’t need just one model.

I might want one model for coding, another for reasoning, one verified for tool calling, one for embedding and a last one for reranking while another I’m experimenting with.

My GPU can run them.

It just can’t keep all of them loaded at once.

This Became a Scheduling Problem

Initially I was thinking about inference. Which model? Which quantization? GGUF or SafeTensors? llama.cpp or vLLM?

But the bigger problem turned out to be scheduling: with only 12GB, deciding which model gets the GPU right now, and getting out of the way of the one that's leaving.

Which model? Which quantization? GGUF or SafeTensors? llama.cpp or vLLM?

But eventually the bigger problem became:

flowchart LR
    A["Request<br/>requested model"] --> B{"Requested model<br/>already loaded?"}
    B -->|Yes| G["Serve request"]
    B -->|No| C["Drain<br/>current model"]
    C --> D["Unload<br/>current model"]
    D --> E["Free<br/>GPU / VRAM"]
    E --> F["Load<br/>requested model"]
    F --> G

I wanted the request to determine which model occupies the GPU.

And I didn’t want every application to understand my inference infrastructure.

Instead of:

Coding model → :8081
Reasoning model → :8082
vLLM model → :8000
Other model → :8083

I wanted:

flowchart TB
    OC["opencode"]
    AG["AI Agents"]
    SDK["Python / Node SDKs"]
    APP["Applications"]

    OC --> API
    AG --> API
    SDK --> API
    APP --> API

    API["One OpenAI-Compatible API<br/>:9090"]

    API --> SW["GGUF Switchboard<br/>Model Scheduler"]

    SW --> LC["llama.cpp"]
    SW --> VL["vLLM"]

    LC --> GG["GGUF Models"]
    VL --> ST["SafeTensors Models"]

When a request comes in for a model that isn't resident, Switchboard drains the current model — it stops accepting new requests and lets in-flight ones finish — then unloads it, waits for VRAM to actually free, and loads the requested model before serving. On my 3060 a swap between 7–12B GGUF models costs a few seconds of cold-load latency, so the scheduler batches consecutive requests for the same model and only pays that cost on an actual switch.

The clients should be boring.

The infrastructure should handle the complexity.

Why Not Ollama or LM Studio?

Both are excellent tools.

Ollama makes downloading and running local models incredibly easy.

LM Studio is excellent for discovering, configuring and experimenting with them.

My requirement was slightly different.

I wanted to treat multiple local models as shared infrastructure, especially on machines where VRAM is constrained.

I also didn’t want to choose one inference ecosystem.

On my 3060, I often prefer:

GGUF → llama.cpp

On larger GPU systems, I may want:

SafeTensors → vLLM

The application shouldn’t care.

It should ask for a model and let the infrastructure determine how to serve it.

So I Built GGUF Switchboard

That experiment became GGUF Switchboard.

It’s an open-source model scheduler that provides one OpenAI-compatible endpoint while managing local model lifecycles underneath it.

Some of the problems I’m trying to solve:

• hardware-aware model discovery
• GGUF + SafeTensors support
• llama.cpp + vLLM backends
• request-driven model switching
• drain/unload/load lifecycle
• GPU resource management
• tool-call conformance testing for AI agents

For example:

ggs models search "qwen"

Instead of just searching for Qwen models, Switchboard considers the hardware it’s running on so I can answer the question I actually care about:

What makes sense on this machine?

The Goal

The simplest way I can describe the project is:

Ollama and LM Studio make running a local model easy. GGUF Switchboard is trying to make operating a collection of local models boring.

For constrained GPUs, that’s exactly what I wanted.

Try It

• GGUF Switchboard is open source: GitHub — GGUF Switchboard

• Demos, screenshots, architecture and documentation: GGUF Switchboard Product Page
  

• For the deeper technical write-up: Read the full article on my blog

• I’ve also published a shorter version on Medium: Running Local AI Coding Models on an RTX 3060 12GB

I’m Looking for Real-World Configurations

If you’re running local LLMs on an 8–16GB GPU, I’d like to know what actually works for you day-to-day:

• GPU
• model + quantization
• llama.cpp / Ollama / LM Studio / vLLM
• coding agent
• tool-calling reliability

If GGUF Switchboard looks useful, try it and star the repo.

More importantly, open an issue if your hardware/model combination doesn’t work.

Real-world configurations are exactly what I want to use to improve the project.