People ask "is a 3090 as fast as an L40S?" and want one number. I measured both on my own rented machines, same script, and there is no one number. It depends on how many requests you run at once.
Setup: vLLM 0.27.1, Qwen2.5 1.5B and 7B (plain and AWQ), 128 output tokens per request, CUDA graphs on, batch 1, 8, 32 and 128, 3 repeats each, median used. RTX 3090 on vast.ai (Netherlands, 24 of 48 vCPUs). L40S on vast.ai (Romania, 8 of 64 vCPUs). The three repeats of each setting landed within 0.5% of each other on the 3090 and within 2.6% on the L40S (worst case), so these are not noise.
Speed: 3090 as a multiple of the L40S
7B fp16:
batch 1 - 1.03x
batch 8 - 1.04x
batch 32 - 1.11x
batch 128 - 0.72x
7B AWQ:
batch 1 - 1.09x
batch 8 - 1.01x
batch 32 - 0.71x
batch 128 - 0.46x
1.5B fp16:
batch 1 - 1.09x
batch 128 - 0.70x
At small batches the two cards are about even. That is roughly what memory bandwidth says (936 vs 864 GB/s). At batch 128 the L40S pulls far ahead, because the work stops being about memory and starts being about compute, and the L40S has much more of it.
So "the 3090 is 1.08x an L40S" is true at batch 1 and wrong at batch 128.
Cost per million output tokens
At the prices I actually paid ($0.414/h for the 3090, $0.826/h for the L40S), counting only time the GPU was busy:
7B fp16:
batch 1 - 3090 $2.22, L40S $4.59
batch 128 - 3090 $0.041, L40S $0.059
7B AWQ:
batch 1 - 3090 $0.77, L40S $1.67
batch 128 - 3090 $0.040, L40S $0.036
At low batch the 3090 is about half the cost per token. At batch 128 the gap closes, and for 7B AWQ the L40S is already cheaper. Which card is cheaper depends on your traffic, not on the card.
Two things this does not tell you
One card is one card. A different 3090 I measured earlier was clearly slower on every setting, with the same script and driver. If you are choosing hardware, measure two units, not one.
And every cost above assumes the GPU is busy. A real server is not. If it serves requests 30% of the time, the real cost per token is about 3.3x these numbers. That idle share can be the biggest number in the bill, and many dashboards don't show it.
The data
All 192 runs, the exact script, and the machine details are public:
https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/3090-own
https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/l40s-own
If you run vLLM and want to see your own idle share, there is a 2-minute guide here (one file, reads only /metrics):
https://github.com/qaisermehdi3-coder/qvunex/blob/main/TRY-IT.md
If you want this measured on your own setup, I take it on as paid work: qvunexaudit@gmail.com
