My desktop runs two mismatched GPUs: a 20GB Ampere card and a 16GB Pascal Tesla. For months, a 21GB vision-language model ran split across both at full GPU speed. Then a routine upgrade of the inference server dropped Pascal support behind a driver-version gate, demoted the old card to a Vulkan device the scheduler refuses to mix with CUDA, and silently rescheduled my model to one GPU plus CPU spill.
Nothing errored. The model still answered. It just answered at 20 to 40 seconds per item instead of 8, which turned a 12-hour benchmark into a 60-hour one. The only evidence was a log line saying the driver was "too old" for a card that had been running CUDA workloads that same morning.
The gate is policy, not physics
The interesting discovery came from reading the server's own startup logs: the bundled CUDA kernels still listed the Pascal compute capability in their build targets. The kernels existed. The Go-side scheduler was refusing to use them based on a driver-version check, while the C++ inference runtime underneath had no such opinion.
That asymmetry is the exploit. The upgraded server ships its actual inference engine as a standalone binary, a vendored build of llama-server, with its CUDA backend as a dynamically loaded library sitting in a subdirectory. The driver gate lives entirely in the scheduler process that launches it.
So: launch the vendored binary directly.
Two mechanics matter. First, the dynamic backend is not found automatically because it lives in a subdirectory the loader does not scan; it needs an explicit environment variable pointing at the .so file itself, not the directory. Get that wrong and the binary silently falls back to CPU while still accepting requests, which cost me one confusing benchmark restart. Second, the vendored build understands the server's own single-file model format, weights and vision projector packed together, which upstream llama.cpp of the same vintage refuses to load. The vendored binary logged "detected combined format, translating" and just worked.
Tensor splits are a policy decision too
With both GPUs visible again, the default split put layers proportional to total VRAM. That is wrong for mismatched cards: the Ampere card is roughly four times faster per layer, so every layer on the Pascal card costs four on the fast one. Skewing the split heavily toward the fast card, leaving the slow card holding only what does not fit, took per-item latency from 17 seconds to 9. Combined with a bigger prefill batch, the full benchmark ran in 12 hours instead of a projected 60-plus.
The general shape of the fix:
- The scheduler said no; the runtime said yes. When a managed serving layer refuses your hardware, check whether the engine underneath actually shares the objection.
- Point the backend loader at the exact library file and verify placement with nvidia-smi before trusting any run. A server that fell back to CPU serves identical responses, slower.
- On mismatched GPUs, split by throughput, not by capacity. Free VRAM on a slow card is not free.
The clean fix is a driver upgrade, and it is scheduled. But the bypass took forty minutes including the two failed attempts, needed no root, touched nothing system-wide, and taught me more about the serving stack than a year of it working silently ever did. The scheduler is there to protect the average user from edge cases. If you can read its logs, you are allowed to disagree with it.
