TensorRT INT8 vs FP16 on RF-DETR, Measured on an RTX 5090

Originally published on khadim.tech.

In short: On an RTX 5090 with TensorRT 10.12, a calibrated INT8 build of RF-DETR put zero layers in INT8. TensorRT's own timing chose FP16 for every layer, so the INT8 engine was the FP16 engine: about 5x faster than the PyTorch FP32 forward, with no extra gain. Forcing INT8 through Q/DQ nodes made it 17% slower and wrecked accuracy. On a CPU without VNNI, only PyTorch dynamic INT8 sped the forward up (1.3-2x), and it changed some detection decisions. Distilling RF-DETR Medium into Nano with pseudo-labels lowered accuracy on every seed. For RF-DETR on an RTX 5090, TensorRT FP16 was the answer, and NVIDIA's own release notes point the same way for DETR models.

I maintain a training and serving engine for RF-DETR defect-detection models. When a model has to meet a cycle time on a given device, the first question is whether it can run faster, so I added a job that builds every faster variant of a trained model (TensorRT FP16 and INT8, CPU INT8 in two runtimes, and a distilled smaller student) and measures each one against the original on the same hardware and the same test images.

Most of the variants that are supposed to be faster were not. These are the measurements, the reasons, and what I recommend instead. The models and their training data stay private; the setup, the method and the ratios below carry over to any RF-DETR model.

How the INT8, FP16 and distillation variants were measured

Measured on 12 October 2026:

Each variant replaces only the network forward. rfdetr calls model.model.inference_model when a model is marked as optimized, so preprocessing, the decoder and the postprocessing are the same code for every variant, and any difference comes from the forward alone. The Nano measurements were repeated three times: forward medians stayed within about 6% on the GPU and 10% on the CPU, and accuracy was identical every time.

The test set is small, so read the accuracy changes as a direction, not a precise figure. The speed numbers are solid.

Why is TensorRT INT8 not faster than FP16 for RF-DETR?

Because TensorRT didn't use INT8. I built the INT8 engine the standard way: INT8 flag, an entropy calibrator over 64 training images, FP16 allowed as the fallback. Then I counted the precision of every layer in the finished engine:

With INT8 allowed, TensorRT times the candidate kernels for each layer and keeps the fastest. It kept FP16 every time, so the "INT8" engine was the FP16 engine with a different name. FP16 cost almost nothing in accuracy: mAP50:95 moved by 0.002 or less, and one decision changed on the segmentation model.

This is a known pattern for DETR models, not an RF-DETR quirk. NVIDIA's TensorRT 10.2 release notes list it as a known issue: "For some Transformer models, including ViT, Swin-Transformer, and DETR, there is a performance drop in INT8 precision (including both explicit and implicit quantization) compared to FP16 precision."

Forcing the issue didn't help. With explicit Q/DQ nodes in the ONNX graph, TensorRT put 166 layers in INT8. That engine was 17% slower than FP16, and mAP50:95 dropped by more than three quarters. I left it out of the job.

The calibrator route is going away

The same release notes deprecate implicit INT8 quantization and the calibrator APIs, IInt8EntropyCalibrator2 included, and NVIDIA's 10.x-to-11.x migration guide says implicit quantization is not supported in TensorRT 11. On TensorRT 11, INT8 means explicit quantization with Q/DQ nodes, the route that was slower here.

Check the precision TensorRT actually chose

An engine that builds with the INT8 flag is not proof that anything runs in INT8. Build with detailed profiling verbosity, then ask the engine inspector:

import json
import tensorrt as trt

# at build time: config.profiling_verbosity = trt.ProfilingVerbosity.DETAILED
inspector = engine.create_engine_inspector()
info = json.loads(inspector.get_engine_information(trt.LayerInformationFormat.JSON))
for layer in info["Layers"]:
    print(layer["Name"], layer["Outputs"][0]["Format/Datatype"])

Without detailed verbosity the inspector returns layer names only, which looks like an empty result rather than an error.

Does the network forward even matter on the GPU?

Barely, in this pipeline. The test frames are 16 megapixels, and preprocessing them for the model took most of each request: the TensorRT forward was between 1% and 5% of a request's time on the GPU host, decoding not even counted. Shrinking the network further would change nothing a user of the line notices. Profile the whole request before optimizing the model.

Does INT8 speed up RF-DETR on a CPU?

One method did, with a cost. I tried two:

PyTorch dynamic INT8 (torch.ao.quantization.quantize_dynamic on every nn.Linear) is a load-time transform with nothing extra to ship. It sped up the network forward by 1.3 to 2.0 times and the whole request by 1.2 to 1.3 times. Accuracy moved only slightly, between −0.014 and +0.009 mAP50:95, but 1 to 4 detections per model crossed the 0.5 threshold one way or the other on 10 images. In inspection, a changed decision is a missed defect or a false reject, so that matters more than the mAP change.

ONNX Runtime static INT8 (per-channel weights, MinMax calibration) was slower than plain PyTorch on this CPU, and its FP32 path was no faster than PyTorch either. It broke the segmentation model whatever the calibration. Percentile calibration helped the detection model (−0.011 instead of −0.026 mAP50:95 on the same images) but needed 30 GB of memory for the Medium model, because ONNX Runtime's histogram calibrators keep every activation of every calibration image in memory.

This CPU has no VNNI, AVX-512 or AMX. INT8 kernels depend on those instructions, so a server CPU that has them could give a different answer; I haven't measured one.

Does distilling RF-DETR Medium into Nano help?

Not with clean labels. The distillation was the simple, hard-label kind. The Medium model predicted on the training images. Every detection at 0.5 confidence or higher that no existing label of the same class already covered (IoU 0.5 or more) became an extra training box. Then two Nano students trained with the same configuration and the same three seeds: one on the labels alone, one on the labels plus the teacher's boxes. Each ran 20 epochs, with no early stopping and no augmentation, and both were scored on the untouched test set.

The student trained with pseudo-labels scored lower on every seed, by 0.025 mAP50:95 on average, and varied more between seeds (standard deviation 0.022 against 0.010). The reason is in what the teacher added. On a fully labelled training set it mostly re-found the existing labels, and it added two boxes across the whole set, both false positives on its weakest class. This kind of distillation can only add value through objects the annotators missed. It is worth trying on incompletely labelled data, and it has nothing to offer on clean labels.

The student was faster: 2.5 times on the CPU and 1.3 times on the GPU. But the Nano and Medium variants have 29.5M and 32.7M parameters, so the speed came from the input resolution (384 px against 576 px), not from a smaller network. If speed is what you need, retrain at a lower resolution and measure that first.

Feature- and logit-based distillation, which train the student on the teacher's internal outputs, weren't part of this test.

How to compare model compression methods fairly

Most of what I learned is about measuring, and it applies to any detector:

• Swap only the forward. Run the same preprocessing, decoder and postprocessing for every variant, or you measure your pipeline changes rather than the compression.
• Report medians. On a shared or virtual host, one stalled call in a hundred can triple a mean.
• Record the host load. One run next to a training job measured a CPU forward more than 50 times slower than its real speed.
• Calibrate on training images, never on the test set. Calibrating on the images you score flatters INT8.
• Count changed decisions at your operating threshold. A small mAP change can hide detections that flip between found and missed.
• Repeat anything that involves training, with three or more paired seeds, before reading a difference.
• Check which precision actually ran, with the inspector above.

What should you use for RF-DETR?

This covers one GPU, one CPU class and three RF-DETR variants. Jetson devices, CPUs with VNNI or AMX, and quantization-aware training for TensorRT 11's explicit INT8 are open questions here. They are the next things I would measure for a customer who needs one of them.

FAQ

Why is TensorRT INT8 not faster than FP16?

With a calibrator and the INT8 flag, TensorRT times INT8 and FP16 kernels per layer and keeps the faster one. For RF-DETR on an RTX 5090 it chose FP16 for every layer, so the INT8 engine was the FP16 engine. NVIDIA's TensorRT 10.2 release notes list INT8 as slower than FP16 for some Transformer models, DETR among them.

Can RF-DETR run in INT8 with TensorRT?

You can force it with Q/DQ nodes in the ONNX graph. In my test that put 166 layers in INT8, ran 17% slower than FP16 and cut mAP50:95 by more than three quarters. A calibrated build with FP16 allowed put no layer in INT8 at all.

Is the TensorRT INT8 calibrator deprecated?

Yes. TensorRT 10.2 deprecated implicit INT8 quantization and the calibrator APIs, IInt8EntropyCalibrator2 included, and NVIDIA's 10.x-to-11.x migration guide says implicit quantization is not supported in TensorRT 11. INT8 there means explicit quantization with Q/DQ nodes.

Does PyTorch dynamic quantization speed up object detection on CPU?

For RF-DETR on an AVX2-only CPU it made the network forward 1.3 to 2.0 times faster and the whole request 1.2 to 1.3 times faster. It also flipped 1 to 4 detection decisions at a 0.5 threshold on a 10-image test set, so check decisions, not only mAP, before shipping it.

Does knowledge distillation make RF-DETR faster?

The distilled Nano student was 2.5 times faster than the Medium teacher on CPU and 1.3 times on GPU, but nearly all of that came from its lower input resolution: the two models have 29.5M and 32.7M parameters. Training it on the teacher's pseudo-labels lowered accuracy on every seed compared with labels alone.

How do I see which layers TensorRT ran in INT8?

Build the engine with detailed profiling verbosity, then read its engine inspector as JSON and look at each layer's output data type. Without detailed verbosity the inspector lists only layer names.

Disclosure: written with AI assistance, from my own measurements and notes.