Same nvJPEG2000, different numbers: timer boundaries and frames in flight

This is an excerpt from a longer write-up, Fastvideo JPEG2000 vs nvJPEG2000 on NVIDIA RTX 4090, which compares two GPU JPEG2000 codecs at the same compressed file size. The part below is about nvJPEG2000 alone: where the timer starts and stops, and how much the same library gives when it is fed differently. Disclosure: the author works at Fastvideo, which makes the other codec in that comparison. The harness, the scripts and the raw logs are open: github.com/fastvideo/jpeg2000-benchmark.

The boundaries in the NVIDIA samples are drawn differently

In NVIDIA's open sample set CUDALibrarySamples the measurement boundaries are drawn differently. Without this distinction, the numbers in the full article and those in NVIDIA's publications appear comparable, but they are not.

The decoding sample. Frames are processed strictly one at a time: one decoder state, one queue of GPU jobs, and a wait after every frame. The -b option, described as a batch size, groups only the reading of files from disk and does not change how the work is done.

The sample measures the running time of a single function, nvjpeg2kDecodeImage. The call is asynchronous: it puts the work into a GPU job queue and returns immediately, so a CPU clock cannot measure it. The sample handles that correctly. The time is taken with a pair of CUDA events on the same queue, before the call and after it, and that is the right way. There is nothing wrong with the measurement of the decoding itself.

The question is about the second term. Parsing of the compressed image is done by nvjpeg2kStreamParse. This is Tier-2, which NVIDIA's own documentation calls the first stage of decoding. Its time is measured separately, with a CPU clock, and added to the total. Here is how that is done in nvjpeg2000DecodeSample.cpp (call arguments omitted, everything else verbatim):

auto io_start = perfclock::now();
nvjpeg2kStreamParse(…);
auto io_end = perfclock::now();
double parse_time = std::chrono::duration_cast<std::chrono::seconds>(io_end - io_start).count();
…
time += static_cast<double>(loopTime / 1000.0);
time += parse_time;

The duration is converted to whole seconds, not to fractional ones. Anything shorter than a second becomes zero, and parsing a frame takes milliseconds. So time += parse_time always adds exactly zero. Allocating GPU buffers, reading the file and writing the result are not part of the measured time, and the finished frame is never copied back to host memory in the measured loop.

This is not a choice of measurement boundary; it is an error in the measurement. What counts as part of the algorithm can be decided in more than one way, and that is a fair argument to have. Here the term is written in the code but is always zero for any frame on any hardware. How much is lost this way can be seen in the stage table in section 8: at decoding, Tier-2 takes from 15 % of the frame time at 2K to 29 % at 4K. Those are fvJPEG2000 shares, because the nvJPEG2000 library does not report time by stage and its own shares are unknown to us. But the stage is the same and runs on the CPU in the same way, so the magnitude is the same.

That is why the nvJPEG2000 numbers in the full article do not come from that sample. A separate program was written for nvJPEG2000 (bench/nvj2k_bench-02/nvj2k_bench-02.cpp, section 14). Its timer starts before nvjpeg2kStreamParse and stops after decoding, once the GPU has finished, so Tier-2 is inside the measured time, exactly as it is for fvJPEG2000. Both sides are measured by the same rule; otherwise there is nothing to compare.

Hence what this means for the reader. The number printed by the NVIDIA sample and the number in the full article cannot be put side by side: the first shows the time of a part of the algorithm, the second covers the whole of decoding. The first will always look better.

The encoding sample. Frames go one at a time there as well. Here the whole per-frame loop is measured, and the copy of the compressed image to host memory (nvjpeg2kEncodeRetrieveBitstream) is inside it. Loading the source frame onto the GPU stays outside, it is done when the file is read. That is exactly the boundary we use in single-frame mode, and there is nothing wrong with it.

Several frames in flight: batching that nvJPEG2000 does not have

An important caveat: batching works differently in the two codecs. This has to be said outright, otherwise the same word in the tables would mean two different things.

First, about what the notation itself means. 8×2 is eight CPU threads, and in each of them two frames are in flight on the GPU at the same time. There are exactly eight CPU threads at any batch size; they do not double. Something else doubles — the number of jobs the GPU computes at the same moment: not eight, but sixteen.

In fvJPEG2000 these two frames go into the codec in a single call: the batch is real, and the codec handles them as one job. This is a standard capability of Fastvideo SDK.

nvJPEG2000 has no such call. Not a single function in the library accepts an array of images — only one image per call. So the GPU load is built up differently: each thread creates as many independent codec states and as many CUDA streams as the batch size specifies. The thread submits encoding of the first frame to its first stream, and immediately after it, without waiting for the result, the second frame to the second stream, and only then waits for both. The calls are asynchronous and the streams are independent, so both frames are computed on the GPU at the same time.

All of this makes use of standard NVIDIA library and CUDA features: multiple codec states, job queues and asynchronous calls are all standard components of both. There are no workarounds here. The only thing missing from the library is a call that accepts several frames at once, so the order of the calls has to be built by hand.

This is also worth saying because it does not work by itself. A program that simply calls nvJPEG2000 one frame per thread — and that is exactly how the NVIDIA samples are built — will get eight simultaneous jobs instead of sixteen, and the result will be lower. How much lower can be seen at one and the same number of threads: on a 2K lossy frame eight threads give the encoder 205 frames per second without the technique and 245 with a batch of two, that is 1.2 times more. For the decoder on that same task the starting point is not measured reliably (section 8), so take the neighboring one: on 4K lossy eight threads give 208 frames per second without the technique and 428 with four frames in flight — twice as much.

We still report exactly these values and take them as the best for nvJPEG2000: the comparison must be against the maximum that can be obtained from the library, not against what the standard way of using it gives.

The standard NVIDIA samples against our own harness

All three columns are the same library doing the encoding — nvJPEG2000 from NVIDIA. Only two things differ: which program calls it, and how that program feeds the frames. On the left, the sample from the CUDALibrarySamples set: the program NVIDIA wrote for its own library and published itself, taken unchanged. In the middle, our own harness calling the same NVIDIA library and processing frames in the same order, one after another; it differs from the left column only in the code around nvJPEG2000 — reading the file, its own timer, the order of the calls. On the right, the same harness and the same NVIDIA library, but with several CPU threads and several frames in flight at once.

The middle column tests the measurement program, not the codec. If the code we wrap around the NVIDIA library kept it from running at full speed, the middle column would come out below the left one — and then the right column would prove nothing, because the gain could be put down to the library never having had a fair chance.

The NVIDIA sample was measured in both of the file-feeding modes it allows, and the better of its results went into the table. In either mode it processes frames strictly in order: one codec state, one CUDA stream, a synchronization after every step.

Both encoders were given the same job, and that is checked rather than assumed. The standard NVIDIA encoder, run with the parameters of section 3.1, produced files of 601,940, 2,966,036, 1,274,517 and 8,964,924 bytes — byte for byte what our own harness produced.

The time boundaries in this part are not the ones used in sections 6 and 7 of the full article, and the results differ accordingly. In all three columns the copying between host memory and GPU memory is outside the measured time: at encoding the source frame is already in GPU memory, at decoding the finished frame stays there. Otherwise the columns would not be comparable — the standard NVIDIA sample only counts what happens on the GPU. In section 7 the boundary is different: the time in multithreaded mode is counted from host memory to host memory, with both copies included.

Everything on this chart is the same library doing the work — nvJPEG2000 from NVIDIA; only the calling program and the way frames are fed differ. The box on the right is how many times our harness with several frames in flight is faster than the same harness feeding frames one after another. The two halves have different scales: decoding 2K lossy reaches 1575 fps while encoding 4K lossless is at 66 fps, and on a common scale half the chart would be unreadable. The values are the ones in the tables below.

Encoding, frames per second; the source frame is already in GPU memory

The best combination of thread count and batch size in the right-hand column: 32×1 on 2K lossy and on both 4K frames, 16×2 on 2K lossless.

The left and the middle column are 1 to 3 % apart. Two different programs on one test system, on the same files, on the same day get the same thing out of the NVIDIA library. So our harness does not keep it from running at full speed, and the whole gain in the right-hand column — 1.2 to 1.5 times — comes from the way the frames are fed, not from the difference between the programs.

Decoding needs a correction before the columns can be compared. The timer of the standard NVIDIA sample covers the parsing of the compressed stream and the decode call, but it measures the parsing in seconds, so the parsing enters its figure as zero. The timer of our harness covers the same region with the parsing counted in full. We measured how far apart they are: both programs were given frames one after another, the frame stays on the card in both cases, and the difference was taken.

At decoding, then, the left column is 3 to 10 % high before any comparison begins, and the gap between it and the middle column is explained by that alone, not by the speed of the library.

Decoding, frames per second; the decoded frame stays in GPU memory

The best combination in the right-hand column: 8×2 on both 2K frames, 32×1 on both 4K frames.

This is where the gap at decoding 2K lossy comes from: 1575 fps here and 1033 fps in section 7, for one and the same library. The copying between GPU memory and host memory accounts for it. A decoded 1920 × 1080 frame, three channels of 8 bits, is 5.93 MB; at the measured bus speed of 25.4 GB/s the copy to host memory takes 0.23 ms. At 1575 fps a frame takes 0.64 ms, so the copy adds a third to it and the limit drops to roughly 1150 fps. The rest goes to copying that does not fully overlap the computation: the measured limit with copying is 1033 fps. The best combination of parameters moves from 8×2 to 8×4 as well, because a larger batch hides the copy time better. Encoding shows no such gap, 291 vs 292 fps: a frame takes 3.4 ms there, and the same 0.23 ms changes almost nothing.

Our own harness speeds up the nvJPEG2000 library itself: 1.2 to 1.5 times at encoding and 1.6 to 5.3 times at decoding. The ratio to take is the one from the middle column to the right-hand one: the same program with the same timer, and only the way of feeding frames differs. The figures are for the best combination of threads and batch found by search (section 4.3); the gain broken down by task is in the table at the top of the article.

Energy per frame — there is no column for the NVIDIA sample here, and this is why. Energy is counted by the card's own meter, by the same difference of two runs. The meter counts the whole time the program is running, and the standard sample re-reads the input files from disk for every batch, while our harness reads a file once. On 4K encoding that is 10.7 seconds of reading out of twenty-four seconds of work: its joules per frame would mostly measure the disk, not the codec. Speed is not affected — there each program's own timer is used, and it does not count the time spent reading files.

Energy of one frame, joules

Both columns are our own harness and the nvJPEG2000 library; only the way of feeding frames differs. Feeding several frames at once saves from 7 % of the energy of a frame at lossless encoding up to a factor of two at 2K lossy decoding: the card does the same work in less time and spends less of it waiting for the next frame.

Energy of one frame by the card's own meter. Both bars in every row are our own harness and the nvJPEG2000 library; only the way of feeding frames differs. The values are the ones in the table above.

The standard NVIDIA samples take no part in the comparison of the two codecs in sections 6 and 7 of the full article. There the nvJPEG2000 library runs under our own harness, that is, in the best shape we could measure it in. A comparison against the standard sample would have made the margin of fvJPEG2000 larger than it is.

Measurement conditions. RTX 4090, driver 610.88, Windows 11, a CPU with 32 logical cores. The compression parameters are the ones of section 3.1: 32×32 code block, six resolution levels, one quality layer, LRCP progression, no tiling. The cells hold the results each program measures for itself. Every point was measured twice — on N frames and on twice as many; in most points the two measurements agree within 1 %, and in two points of the right-hand column they are up to 7 % apart. Every point in these tables ran at the full GPU clock, 2745 to 2760 MHz; the clock was sampled ten times a second throughout each measurement.

This is a separate run, not the one sections 6 and 7 of the full article are made from, so the numbers differ from them by a few percent: the grid of combinations here is shorter. What should be compared is the columns inside these tables — they were measured one after another, under the same conditions. The run script and the full logs are in the repository (section 14).

Excerpt from Fastvideo JPEG2000 vs nvJPEG2000 on NVIDIA RTX 4090. Text under CC BY-ND 4.0, measurement results under CC BY 4.0. Harness, scripts and raw logs: github.com/fastvideo/jpeg2000-benchmark.