Measuring RKLLM Latency on an SBC

A published RKLLM test on an 8GB KiwiPi 5 recorded about 15 tokens/s for Qwen3.5-0.8B. Increasing the input from 39 to 746 tokens pushed the wait for its first text from 0.35 to 4.44 seconds, while decode throughput barely moved.

The experiment compared that model with Qwen3.5-2B on the RK3588S NPU. The full RKLLM setup and benchmarks cover installation, memory and the complete results. What I want to pull out here is the measurement code: a tokens/s counter can hide several seconds your application still has to wait.

Keep loading out of request latency

The test used Ubuntu 24.04.3 LTS, Linux 6.1.141, RKNPU 0.9.8 and RKLLM Runtime 1.3.1. Both model files were pre-converted W8A8 artifacts downloaded for the experiment. The converted Qwen3.5 model repository identifies the 2B artifact and its target hardware.

A request starts after the model has loaded. If you start a process, initialize its model and send a prompt under one stopwatch, you've measured startup plus inference. That may be the right measurement for an application that exits after each answer. A service keeping one model loaded has a different path.

The measurement client recorded initialization separately. For request timing, the clock started immediately before rkllm_run, and stopped for the first-text measurement when a callback supplied its first non-empty text. That's an application-visible event, which is why I'm calling it first text rather than claiming to observe the first internal token.

The callback is the useful boundary

The official Rockchip RKLLM SDK provides the C/C++ Runtime API. The test author used this API to build a small command-line measurement client specifically for the experiment. It runs sequential requests; it isn't an inference server or a released application.

This is the timing portion, adapted from that client's result callback for RKLLM 1.3.1:

#include <chrono>
#include "rkllm.h"

using Clock = std::chrono::steady_clock;
static Clock::time_point request_start;
static double first_text_ms = -1;
static bool inference_failed = false;
static RKLLMPerfStat perf{};

static int on_result(RKLLMResult* result, void*, LLMCallState state) {
    if (state == RKLLM_RUN_ERROR) {
        inference_failed = true;
        return 0;
    }
    if (state == RKLLM_RUN_NORMAL && result && result->text &&
        result->text[0] && first_text_ms < 0) {
        first_text_ms = std::chrono::duration<double, std::milli>(
            Clock::now() - request_start).count();
    }
    if (state == RKLLM_RUN_FINISH && result) {
        perf = result->perf;
    }
    return 0;
}

Register this as RKLLMCallback::result_callback when initializing the handle. Before each synchronous call, reset the state and clear the KV cache. The following excerpt assumes the handle, input and inference parameters have already been initialized:

int clear_rc = rkllm_clear_kv_cache(handle, 1, nullptr, nullptr);
if (clear_rc != 0) return clear_rc;

first_text_ms = -1;
inference_failed = false;
perf = {};
request_start = Clock::now();
int run_rc = rkllm_run(handle, &input, &infer_params, nullptr);

A nonzero return code or inference_failed marks a failed request. If first_text_ms stays negative, no non-empty text callback was recorded. The globals make this short enough to read, but only suit the single-request client here; a concurrent server needs state associated with each request.

Decode throughput came from perf.generate_tokens / (perf.generate_time_ms / 1000.0), with a positive-time check. The calculation used the actual returned count, rather than the configured output limit. The main requests capped generation at 128 new tokens; the Runtime reported 127 generated tokens.

Input length changes the result

These are medians of three independent requests at each input length, with context capacity set to 2048. Thinking and conversation history were disabled, and KV cache was cleared before every request.

The 0.8B decode column stays near 15 tokens/s. A UI built around that number alone would miss the difference between text arriving almost immediately and a four-second pause. If your application resends a document with every question, use inputs resembling that document in the test.

And keep cache policy in the record. Clearing KV makes these independent requests; it doesn't describe a conversation that reuses earlier context. A separate phase raised configured capacity to 4096 while retaining the same 746-token input. That is not a measurement of a full 4096-token prompt.

A successful call can still return a bad answer

All 58 requests in the final series completed without an inference error. That includes performance, output-quality, context-capacity and repeated-request phases, not 58 repetitions of one benchmark prompt.

Both models returned the exact requested JSON object and wrote a simple max(a, b, c) function that passed four limited checks. Both also made unsupported technical claims. The smaller model claimed the device learns from its own experiences; the larger one suggested offloading prevents lag or overheating.

The arithmetic example hit another limit. With 192 new tokens allowed for quality tasks, 2B derived the correct result but stopped before finishing its final sentence. The 0.8B response stopped before reaching a result. Treat an output cut off by the generation budget as incomplete; testing a larger budget is a different run.

The experiment included no CPU-only baseline or power measurement, so these numbers cannot establish an NPU speedup or energy saving. They describe two models running through this Runtime on this board.

For an application benchmark, I'd keep the prompt set fixed, log first text and decode separately, and validate the output before comparing speed. A response that arrives quickly and gets the task wrong still needs fixing.