What PREEMPT_RT Changes: Sleeping Spinlocks (Part 1)

On a PREEMPT_RT kernel, spinlock_t is not a spinlock. It is an rtmutex, and a task that fails to acquire one goes to sleep. The reason the rest of RT looks the way it does is that preempt_disable() was quietly doing three jobs at once, and once RT stops calling it, the other two have to be done by hand. Sleeping spinlocks are where that shows up in the code: rt_spin_lock() takes the lock, then calls rcu_read_lock() and migrate_disable() to put back what preemption disabling used to provide for free.

Most explanations of PREEMPT_RT start with latency numbers. This series starts with the source, because the numbers follow from a handful of substitutions and the substitutions are more interesting than the histograms. This first part takes the central one: sleeping spinlocks, the conversion of spinlock_t into something that can block.

What preempt_disable() was actually buying you

On a mainline kernel, spin_lock() disables preemption and then spins on an atomic variable until the lock is free. Ask why preemption is disabled and the obvious answer is deadlock avoidance: if the holder could be preempted by another task on the same CPU that wants the same lock, that task would spin forever against a holder that is not running.

That is correct, and it is not the whole answer. Disabling preemption also has two side effects that a great deal of kernel code silently depends on.

The first is CPU pinning. A task that cannot be preempted cannot be migrated, so any pointer it holds into per-CPU data stays valid for the whole critical section. Code that calls this_cpu_ptr() inside a spinlock and expects the result to still refer to its own CPU three lines later is relying on this.

The second is RCU. A non-preemptible region is, on a classic RCU configuration, implicitly an RCU read-side critical section, because a grace period cannot complete while a CPU sits in one. Code that dereferences an RCU-protected pointer under a spinlock without calling rcu_read_lock() is relying on this too.

The authors of the RT substitution say this in the file header, which is worth reading before any of the code:

/*
 * PREEMPT_RT substitution for spin/rw_locks
 *
 * spinlocks and rwlocks on RT are based on rtmutexes, with a few twists to
 * resemble the non RT semantics:
 *
 * - Contrary to plain rtmutexes, spinlocks and rwlocks are state
 *   preserving. The task state is saved before blocking on the underlying
 *   rtmutex, and restored when the lock has been acquired. Regular wakeups
 *   during that time are redirected to the saved state so no wake up is
 *   missed.
 *
 * - Non RT spin/rwlocks disable preemption and eventually interrupts.
 *   Disabling preemption has the side effect of disabling migration and
 *   preventing RCU grace periods.
 *
 *   The RT substitutions explicitly disable migration and take
 *   rcu_read_lock() across the lock held section.
 */

That is the design of sleeping spinlocks in one comment. RT cannot disable preemption, because the whole point is to keep the kernel preemptible. So it must replace the mutual exclusion with something that blocks, and then re-create the two side effects explicitly.

Sleeping spinlocks are invisible at the call site

Nothing in a driver changes. There is no #ifdef CONFIG_PREEMPT_RT at the usage site, because the substitution happens in the headers. include/linux/spinlock_rt.h opens by refusing to be included on its own:

#ifndef __LINUX_INSIDE_SPINLOCK_H
#error Do not include directly. Use spinlock.h
#endif

and then redefines the API in terms of an rtmutex. Initialisation makes the substitution explicit:

#define __spin_lock_init(slock, name, key, percpu)
do {
    rt_mutex_base_init(&(slock)->lock);
    __rt_spin_lock_init(slock, name, key, percpu);
} while (0)

A spinlock_t under RT contains an rt_mutex_base, and initialising the spinlock initialises the rtmutex inside it. The documentation states the two halves of the rule plainly: on a non-PREEMPT_RT kernel spinlock_t is mapped to raw_spinlock_t and has exactly the same semantics, while on a PREEMPT_RT kernel it is mapped to a separate rt_mutex-based implementation. raw_spinlock_t is a strict spinning lock in all kernels, including RT ones. The split between the two types is the entire design, and it is why RT required touching so much of the tree: every lock had to be classified as one that may sleep or one that must not.

What rt_spin_lock() actually does

The implementation is short enough to read in full, and every line of it is the argument above made concrete.

static __always_inline void rtlock_lock(struct rt_mutex_base *rtm)
{
    lockdep_assert(!current->pi_blocked_on);

    if (unlikely(!rt_mutex_cmpxchg_acquire(rtm, NULL, current)))
        rtlock_slowlock(rtm);
}

static __always_inline void __rt_spin_lock(spinlock_t *lock)
{
    rtlock_might_resched();
    rtlock_lock(&lock->lock);
    rcu_read_lock();
    migrate_disable();
}

Four things happen, in order. rtlock_might_resched() is the debug annotation that this call may block. rtlock_lock() tries a single compare-and-exchange to install the current task as owner, and falls into rtlock_slowlock() only on contention — so an uncontended acquisition on RT is one atomic operation, much like the mainline fast path. Then rcu_read_lock() and migrate_disable() restore the two side effects that were lost.

This is the part worth remembering about sleeping spinlocks. The lock itself is the least surprising piece. The interesting work is the two lines that put back what preempt_disable() used to provide without anyone asking for it. Your code that dereferenced an RCU pointer under a spinlock still works on RT, not because RT is forgiving, but because rt_spin_lock() called rcu_read_lock() on your behalf. Your this_cpu_ptr() is still valid for the same reason: the task is pinned, by migration disabling rather than by preemption disabling, so it remains preemptible while staying on its CPU.

The suffixes sleeping spinlocks quietly ignore

This is where sleeping spinlocks surprise people reading RT code for the first time. Consider the interrupt-related variants:

static __always_inline void spin_lock_irq(spinlock_t *lock)
{
    rt_spin_lock(lock);
}

#define spin_lock_irqsave(lock, flags)           \
    do {                                         \
        typecheck(unsigned long, flags);         \
        flags = 0;                               \
        spin_lock(lock);                         \
    } while (0)

spin_lock_irq() does not disable interrupts. spin_lock_irqsave() does not save them either; it assigns zero to your flags variable and takes the lock. The matching spin_unlock_irqrestore() ignores the flags argument entirely and calls rt_spin_unlock(). This has to be so: an rtmutex acquisition can block, and blocking with interrupts disabled is not a thing the kernel can do.

The bottom-half suffix is the exception, and it is a real one:

static __always_inline void spin_lock_bh(spinlock_t *lock)
{
    /* Investigate: Drop bh when blocking ? */
    local_bh_disable();
    rt_spin_lock(lock);
}

Softirq handlers really are still disabled. The comment left in the source is an honest one about an unresolved question, and it is the kind of thing you only find by reading the file.

Two more of these are worth knowing. spin_is_contended() is defined as (((void)(lock), 0)) — it always reports no contention, so any adaptive logic you have that backs off when a lock is busy quietly stops adapting. And spin_trylock_bh() has to undo its own work on failure, disabling bottom halves, attempting the lock, and re-enabling them if the attempt did not succeed.

Why the unlock order is what it is

The release path looks over-specified until you read the comment attached to it.

void __sched rt_spin_unlock(spinlock_t *lock)
{
    spin_release(&lock->dep_map, _RET_IP_);
    migrate_enable();

    if (unlikely(!rt_mutex_cmpxchg_release(&lock->lock, current, NULL)))
        rt_mutex_slowunlock(&lock->lock);

    rcu_read_unlock();
}

Migration is re-enabled first, then the lock is released, and rcu_read_unlock() is deliberately last. The source explains why with a use-after-free trace: a second task can be sitting in an RCU read-side section holding a pointer to the same object, waiting on the same lock. If rcu_read_unlock() ran before the release, the grace period could end while the unlocking task is still about to touch lock->lock, a kfree_rcu() callback could run, and the compare-and-exchange would then write into freed memory.

The RCU read-side section that rt_spin_lock() took as a convenience for callers turns out to also protect the lock word itself. That is the kind of constraint that only exists because sleeping spinlocks made the lock a real object with a lifetime, rather than a word you spin on.

How sleeping spinlocks avoid losing a wakeup

A plain rtmutex sets the blocking task to TASK_UNINTERRUPTIBLE. Sleeping spinlocks cannot do that, because the caller may already be in a carefully chosen state — a driver that has set TASK_INTERRUPTIBLE and is about to sleep would find its state destroyed by taking a lock.

So the RT lock path saves and restores it, using current_save_and_set_rtlock_wait_state() before blocking and current_restore_rtlock_saved_state() on acquisition, with a dedicated TASK_RTLOCK_WAIT state and a schedule_rtlock() variant. The documented sequence is:

task->state = TASK_INTERRUPTIBLE
lock()
     block()
       task->saved_state = task->state
       task->state = TASK_UNINTERRUPTIBLE
       schedule()
                                      lock wakeup
                                        task->state = task->saved_state

The subtle case is a real wakeup arriving while the task is blocked on the lock. It cannot be delivered, because the task must stay blocked until the lock is free. Instead the wakeup writes TASK_RUNNING into saved_state, and the lock wakeup later restores that, so the task ends up runnable and the wakeup is not lost.

What this means for code you write

The practical rules follow from how sleeping spinlocks are built rather than being a separate list to memorise.

Anything that must not sleep must use raw_spinlock_t. That is low-level interrupt handling, scheduler and timer core, and places where hardware state is being touched. The documentation also permits it where a critical section is genuinely tiny, to avoid rtmutex overhead.

Inside a raw_spinlock_t you may not take a spinlock_t, because the outer lock disabled preemption and the inner one may sleep. The nesting order is fixed: sleeping locks, then spinlock_t and rwlock_t and local_lock, then raw_spinlock_t and bit spinlocks. Lockdep enforces this on RT and non-RT kernels alike, which is the single most useful fact in this article: you can find most RT violations on a mainline kernel by building with lockdep enabled, long before you boot an RT kernel.

Allocation flips in a way that surprises people. Calling kmalloc(GFP_ATOMIC) under a raw_spin_lock() fails on RT, because the allocator is fully preemptible and cannot be called from a truly atomic context. Calling it under an ordinary spin_lock() is fine on RT, precisely because that lock no longer disables preemption.

And do not pair interrupt disabling with a spinlock_t by hand. Writing local_irq_disable() followed by spin_lock() is correct on mainline and broken on RT, because the rtmutex needs a preemptible context. Use spin_lock_irq() or spin_lock_irqsave(), which do the right thing in both configurations even though, as shown above, they do almost nothing on RT.

One type cannot be converted at all. A bit spinlock is a single bit, and a bit is too small to hold an rtmutex, so bit spinlocks keep their spinning semantics on RT and inherit every raw_spinlock_t restriction. Where that was unacceptable, the conversion had to be done with conditional code at the usage site rather than in the headers.

Key takeaways

• Under PREEMPT_RT, spinlock_t maps to an rtmutex and may block. Sleeping spinlocks are the result; raw_spinlock_t remains a true spinning lock in every configuration.
• preempt_disable() was doing three jobs: excluding preemption, pinning the CPU, and blocking RCU grace periods. Sleeping spinlocks replace the first with a blocking lock and restore the other two by calling rcu_read_lock() and migrate_disable().
• The _irq and _irqsave suffixes do not touch the interrupt state on RT; spin_lock_irqsave() assigns zero to your flags. The _bh suffix still disables softirqs.
• spin_is_contended() always returns 0 on RT, so back-off logic built on it stops working.
• rt_spin_unlock() calls rcu_read_unlock() last on purpose, because that read-side section is also what keeps the lock word itself alive against a concurrent kfree_rcu().
• The nesting rule — no spinlock_t inside a raw_spinlock_t — is enforced by lockdep on mainline too, so most RT locking bugs are findable without an RT kernel.

Frequently asked questions

Does my driver need changes to work with sleeping spinlocks?
Usually not. The substitution happens in the headers, so there is no conditional code at the call site and spin_lock() keeps its name and signature. What needs review is any place you nested a spinlock_t inside a raw_spinlock_t, disabled interrupts by hand around a lock, or relied on spin_is_contended().

If spinlock_t can sleep, why is an uncontended lock still fast?
Because the fast path is a single compare-and-exchange. rtlock_lock() attempts rt_mutex_cmpxchg_acquire() to install the current task as owner and only calls rtlock_slowlock() when that fails, so an uncontended acquisition does not schedule.

Why does spin_lock_irqsave() still exist on RT if it does nothing?
So that one source file compiles and behaves correctly in both configurations. On mainline it saves and disables interrupts; on RT it sets your flags variable to zero and takes the rtmutex. Writing local_irq_disable() and spin_lock() separately is what breaks, because the rtmutex needs a preemptible context.

Can I call kmalloc() while holding a lock on an RT kernel?
Under an ordinary spin_lock(), yes, because that lock no longer disables preemption. Under raw_spin_lock(), no, not even with GFP_ATOMIC, because the allocator is fully preemptible and cannot run in a truly atomic context.

How do I find RT locking violations without an RT kernel?
Build with lockdep enabled. The nesting constraints between sleeping locks, spinlock_t and raw_spinlock_t are checked on PREEMPT_RT and non-PREEMPT_RT kernels alike, so a mainline kernel with lockdep will report most of these before you ever boot RT.

What is next in this series

Sleeping spinlocks rest on the rtmutex, so Part 2 takes that: how priority inheritance works, what the chain walk does when a boosted task is itself blocked, and why a counting semaphore cannot be given the same treatment. Later parts cover forced interrupt threading, local_lock and migrate_disable(), softirqs and RCU under RT, and the timer and console changes.

Further reading

• Lock types and their rules — the kernel's own description of the three lock categories, the PREEMPT_RT caveats and the nesting order.
• kernel/locking/spinlock_rt.c — the substitution itself, including the file header comment and the use-after-free note in rt_spin_unlock().
• include/linux/spinlock_rt.h — the API mapping, where the suffix behaviour is visible.
• RT-mutex implementation design — background for Part 2.

Originally published at techveda.live.