optimizing rust async executors for sub‑millisecond latency

in a high‑frequency trading gateway we hit a hard 1 ms SLA for order‑book updates. the naive tokio default (multi‑threaded, work‑stealing) added ~300 µs of jitter because each task hopped between cores and suffered cache line ping‑pong.

The bottleneck

the OS scheduler tick (typically 1 ms on Linux) and the executor’s global queue are the two dominant sources of latency. when a task is woken it lands in a lock‑protected queue; the next worker thread may not see it until the next tick, inflating tail latency. this is a hardware‑level constraint: each core’s L1 cache must be refreshed, and a lock incurs a memory fence that serializes execution.

A leaner executor

building a single‑threaded, poll‑driven executor removes the lock and keeps the whole pipeline on one core, preserving cache locality. the trade‑off is reduced parallelism, so it only makes sense when the workload is I/O‑bound and can be multiplexed on a single core.

use std::future::Future;
use std::task::{Context, Poll};
use std::pin::Pin;

struct SpinExecutor {
    tasks: Vec<Pin<Box<dyn Future<Output = ()> + Send>>>,
}

impl SpinExecutor {
    fn new() -> Self { Self { tasks: Vec::new() } }
    fn spawn<F>(&mut self, f: F) where F: Future<Output = ()> + Send + 'static {
        self.tasks.push(Box::pin(f));
    }
    fn run(&mut self) {
        while !self.tasks.is_empty() {
            let mut i = 0;
            while i < self.tasks.len() {
                let waker = futures::task::noop_waker_ref();
                let mut cx = Context::from_waker(waker);
                match self.tasks[i].as_mut().poll(&mut cx) {
                    Poll::Ready(_) => { self.tasks.swap_remove(i); },
                    Poll::Pending => i += 1,
                }
            }
        }
    }
}

fn main() {
    let mut exec = SpinExecutor::new();
    exec.spawn(async { /* low‑latency I/O */ });
    exec.run();
}

The snippet shows a lock‑free, busy‑wait loop. in production you’d add a timerfd or epoll wait to avoid 100 % CPU, but the core idea stays: keep the task queue in thread‑local memory.

Trade‑offs and failure modes

• CPU utilization – a pure spin loop can saturate a core; you need a back‑off strategy or integrate with epoll to sleep when idle.
• Scalability – moving to N cores re‑introduces contention; hybrid models (per‑core queues + work‑stealing only for overflow) often hit the sweet spot.
• Back‑pressure – without a bounded queue a burst of network packets can OOM the process; enforce a limit and drop or throttle early.
• Debugging – standard tools (tokio console) assume the Tokio runtime; with a custom executor you lose that visibility and must instrument manually (e.g., perf record).

Real‑world results

on a 12‑core Xeon we measured a 40 % reduction in 99th‑percentile latency (from 1.2 ms to 720 µs) after switching to a per‑core spin executor and pinning the thread to a single CPU. the trade‑off was a modest 5 % drop in overall throughput, which the business accepted because tail latency directly impacted revenue.

anyone tackled sub‑millisecond latency with a custom async runtime on multi‑core hardware? how did you balance the CPU cost of spinning with the need for deterministic response times