RDMSR

FreeBSD GUS

2026-10-02T00:00:00+00:00

Introduction

FreeBSD has a pretty neat safe memory reclamation scheme (something akin to RCU), that's also been adopted by XNU that hasn't really been well-documented outside of code comments, and I thought it would've been interesting to write about it.

What safe memory reclamation is all about

The problem of safe memory reclamation (SMR) shows up notably when working with lock-free data structures. Lock-free data structures are the bread and butter of modern scalability in operating systems, and they often allow much higher throughput than their finely grained locked counterparts, but they come with their own issues.

Lock-free data structure code in unmanaged languages needs to carefully manage the liveness of the objects it manages, i.e it must guarantee that memory is freed if and only if no one is able to access it. This arises because there is no synchronization around the access of the data structure, so an object that is removed can still be visible concurrently by another CPU (who might have made a local copy).

More broadly, SMR schemes can also be used for other things than memory reclamation, I like to think of them as providing barriers ensuring that all participants (read: CPUs or threads) have passed through a certain point.

Prior art

SMR schemes broadly fall in two families: Epoch-based reclamation (EBR) and Quiescent-state based reclamation (QSBR). You might have also heard of Hazard Pointers (HP), which are useful on their own but are seldom seen in operating systems (at least, that I know of!).

QSBR is notable because you might've heard of it called a different name by the Linux project, RCU (Read-Copy-Update)! RCU's author, Paul McKenney, actually invented QSBR for the DYNIX operating system.

RCU has been described extensively, and I would recommend reading McKenney's excellent LWN Articles to learn more. The TL;DR is that there are three operations: rcu_read_lock, rcu_read_unlock and rcu_synchronize, the read operations are used to guard reads around potentially-freed memory (i.e to enter a read section) and the synchronize function waits until there is no CPU in a read section (this could be used to wait until memory is safe to be freed).

The gist of the algorithm is that memory is known to be safe to free when all CPUs have gone through at least one quiescent point ever since the object was removed from its data structure (a grace period), and read section operations block those quiescent points from ever happening.

In practice, this can be implemented by having a quiescent point on context switches and inhibiting preemption during read operations. The core of the algorithm is actually pretty simple, and much of the complexity of the Linux code stems from making this scalable (Tree RCU) and handling other RCU flavors (e.g preemptible RCU).

FreeBSD's safe memory reclamation scheme is called GUS and is part of the EBR family.

Epoch-based reclamation

Epoch-based reclamation was first described by Keir Fraser in his now-famous thesis titled Practical Lock-freedom (link). The way it works is that there is a global epoch which moves through 0, 1 and 2. When a thread enters a read section, it pins itself by reading the global epoch and copying it into its local slot.

When a writer retires an object, it goes into a "limbo" bag tagged with the current epoch (really, the epoch is used as an index into an array of three linked lists). The global epoch is advanced by polling all other threads and checking whether or not they have acknowledged the current epoch by reading their local slot; if it so happens that they have, then the global epoch is advanced and memory that was retired two epochs ago can now be freed.

The main issues that arise:

Epochs are bounded, i.e they always cycle through 0, 1 and 2, this means that writers have to synchronize with epoch advancement before retiring more objects; and
Doing an O(n) scan touching remote memory is very expensive!

It is now time to reveal that GUS really means Global Unbounded Sequences, and it tackles on issue #1 by making epochs unbounded (hence the name) and issue #2 is somewhat mitigated by clever integration with the broader system.

GUS

I am using sequence and epoch interchangeably here. Confusing, I know!

The way GUS works is that there is a global, monotonically increasing write sequence, as well as a global read sequence, which represents the last minimum observed sequence across CPUs.

Each CPU also keeps track of its own local sequence when it is pinned, akin to how EBR does things.

The algorithm goes as follows, in pseudocode:

enter():
    store(cpu.cur_seq, load(global_write_seq, relaxed), relaxed)
    fence()

exit():
    store(cpu.cur_seq, SEQ_INVALID, release)

advance():
    fetch_add(global_write_seq, 1, release)

// Returns whether a sequence `goal` is reached
poll(goal):
    if goal <= global_read_seq:
        // Goal already reached!
        return true

    min_seq = global_read_seq

    for each cpu:
        min_seq = min(min_seq, cpu.cur_seq)

    if goal <= min_seq:
        // Goal reached across all CPUs
        Exchange global_read_seq with the new one.
        return true
        
    return false

The basic idea is therefore that a sequence is essentially a timestamp^[1], and that the read sequence is always playing catch-up with the write sequence, but never actually reaching it, (i.e there is always at least a separation of 1 between both). Obviously, in practice, there is a bit more to it especially w.r.t memory orderings, but the whole thing is still freakishingly simple and can be implemented in a few hundred lines, see the FreeBSD implementation here.

The unbounded epochs solved the issue #1 that was presented earlier, but what about issue #2: how is polling amortized?

Jeff Roberson, the author of GUS, also wrote the UMA (universal memory allocator) implementation, and both tie nicely together.

Integration with the rest of the system

To me, the way GUS is integrated with the allocator is really its distinguishing feature and what makes the whole thing work well.

The UMA allocator is essentially a modernized slab allocator, which was first implemented in Solaris and described in Jeff Bonwick's influential papers^[2]^[3] in the early 2000s. To make things scale on SMP systems, it relies on per-CPU "buckets" (magazines in Bonwickspeak), which are essentially an array of N objects on which objects are popped from and pushed onto quickly. A certain amount of pre-allocated buckets are stored in a per-zone (kmem_cache in Bonwickspeak) "depot" from which empty buckets and full buckets are pulled. N, i.e the size of buckets, is dynamically resized according to measured contention on the depot lock; high contention needs to be reduced by increasing the bucket size, which provides amortization over the (potentially) slow and contended operation of going to the depot.

GUS and the allocator join together by allowing the creation of "SMR zones" which are zones that do careful handling around the freeing of objects. Someone wanting to write a lock-free data structure could create a SMR zone for their object and do allocations and frees effectively transparently.

On SMR zones, when the allocator notices a bucket is completely free (i.e full of objects), it advances the write sequence and stamps the bucket with it, then puts it back at the end of the depot free list.

When the allocator then needs to get a free bucket from the depot, it gets the bucket from the head of the free list, and then polls other CPUs for the sequence that is stamped on that bucket. If the goal has already been reached, then the bucket can be reused, otherwise, the allocator will fallback on allocating another bucket itself.

Diagram of the zone depot

What's cool about this is that buckets naturally provide amortization for those SMR operations; polling is done on a batch of objects at once, and not on a per-object basis (which avoids starting one write sequence per object) and growing the buckets lengthens the time spent before touching the depot again, which gives the time for other buckets to age.

Remember that the GUS algorithm relies on finding the lowest sequence across all CPUs, which means that oftentimes, if buckets are aged as expected, then a single poll operation could allow the reuse of many buckets at the same time, e.g if the depot free list contains buckets stamped with 100, 101, 102 and we notice via polling that the new minimum sequence is 103, then all of those buckets can be reused immediately, which avoids having to do future polls for those buckets.

That is what makes everything click together and the polling not so bad anymore!

XNU

Apple's XNU has also adopted a very similar scheme but with many goodies on top. For instance, they support preemptible read sections and a call_rcu-style API. The way it integrates with their zone allocator, however, is quite interesting and its implementation differs enough that I deem it worthy to be described here as well.

XNU's zone allocator (zalloc) is, like UMA, also essentially a modernized slab allocator; I will not try to explain it in depth here because it is quite sophisticated (over 10k lines!), and I am keeping this in reserve for another post, but I will go over the details of how their buckets are reused.

What's interesting is that zalloc does not rely on resizing buckets at all, instead, buckets are kept at a fixed size (something like 8 objects), and a per-CPU depot is installed on top of the per-zone depot, which they call the "recirculation depot" (this should make sense later why such a name was used). This per-CPU depot is then resized with an algorithm similar to the one used to resize buckets, providing the same kind of amortization.

When a bucket becomes full and ready to be recycled, it is stamped with a deferred advance of the write sequence and put onto the per-CPU depot. When allocation is in need of a bucket, it polls the head of the per-CPU depot, and grabs it if it has expired. Otherwise, it does four things:

Append all the buckets from the per-CPU depot to the zone depot

Since the head of the per-CPU depot hasn't expired yet, this means that all buckets that sit in the per-CPU depot are not yet ready. All buckets are therefore put at the end of the zone depot, as to let them age behind the buckets that were already in depot.

Commit the sequence of the last bucket in the zone depot

The last bucket in the depot is the newest, its sequence is taken and the global write sequence is advanced to that one immediately, this allows to do one write sequence write for many buckets.

Pull back a bunch of buckets from the head of zone depot into the per-CPU depot

A batch of buckets is then pulled from the head of the zone depot, these are the oldest buckets, so they are more likely to have expired. The batching here avoids having to constantly thrash on the depot and avoids contention on it.

Retry to allocate from per-CPU depot

Finally, the oldest bucket we just freshly pulled from the zone depot is polled again, if it still hasn't expired, then there is no choice but to fallback on allocating new memory.

What's all this jazz about deferring and committing? I like to think of a deferred advance as a "fake advance", it snapshots the current sequence but does not increment it, which avoids a global read-modify-write (RMW) operation on the global write sequence. The key idea is that there is no need to advance the sequence, i.e commit it, until we actually need to, and this also serves as an amortization feature because a sequence can be jumped to instantly, for instance, from 1 to 5 instead of individually advancing through 1, 2, 3, 4, and 5.

The recirculation depot is hence called as such because it allows a given bucket to recirculate and age before given the chance to be reused again.

Diagram of the recirculation depot

Pros & Cons

This SMR scheme, like anything else, has its own tradeoffs. It is relevant to compare it with RCU because both are analogous, compared to RCU:

Pros:

Only CPUs actively in a read-section need to participate during polling.
Simple implementation
Lower free-to-reuse latency^[4]

Cons:

Even though polling is amortized, it is still an expensive operation, and might even be a no-go on large NUMA systems.
Higher overhead on entering read sections (needs a fence)

Resources

The best resource, as always, is reading the actual code, which is thankfully very well commented (GUS link, UMA link). The XNU code is also heavily commented (SMR link, zalloc link).

Additionally, these are useful resources:

Jeff Roberson has made a short talk about GUS and UMA.
Comparison with RCU by Joel Fernandes, a Linux developer.

I have also implemented GUS and an allocator inspired by XNU's and UMA in my own operating system, zag (SMR link, allocator link)

Footnotes

In fact, one could use a timestamp counter such as the TSC for this, as seen in Parallel Sequences. ↩
The Slab Allocator: An Object-Caching Kernel Memory Allocator ↩
Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources ↩
This was actually measured by the FreeBSD people, see the talk I mentioned above. ↩

Poor man's traits in Zig

2026-09-09T00:00:00+00:00

Zig is a pretty nice programming language, but it unfortunately does not support compiler-enforced interfaces or traits. Instead, the language relies on duck typing to operate on generic types.

Luckily, Zig supports static reflection and compile-time code evaluation, so it is quite easy to implement "Poor man's traits":

/// Asserts that a given type `T` matches the schema declared by `I`.
/// This includes public methods and fields.
pub fn assert_interface(T: type, I: type) void {
    const tinfo = @typeInfo(I);

    inline for (tinfo.@"struct".fields) |f| {
        if (!@hasField(T, f.name)) {
            @compileError(std.fmt.comptimePrint("Expected field '{s}' of type '{s}' in type '{s}' required by '{s}'", .{ f.name, @typeName(f.type), @typeName(T), @typeName(I) }));
        } else {
            if (@FieldType(T, f.name) != f.type) {
                @compileError(std.fmt.comptimePrint("Expected field '{s}' of type '{s}' in type '{s}' required by '{s}', got '{s}'", .{ f.name, @typeName(f.type), @typeName(T), @typeName(I), @typeName(@FieldType(T, f.name)) }));
            }
        }
    }

    inline for (tinfo.@"struct".decls) |decl| {
        const member = @field(I, decl.name);

        if (@typeInfo(@TypeOf(member)) == .@"fn") {
            if (!@hasDecl(T, decl.name)) {
                @compileError(std.fmt.comptimePrint("Expected method '{s}' in type '{s}' required by '{s}'", .{ decl.name, @typeName(T), @typeName(I) }));
            }

            const impl_member = @field(T, decl.name);
            const IfaceFnType = @TypeOf(member);
            const ImplFnType = @TypeOf(impl_member);

            if (IfaceFnType != ImplFnType) {
                const iface_fn = @typeInfo(IfaceFnType).@"fn";
                const impl_fn = @typeInfo(ImplFnType).@"fn";

                if (iface_fn.params.len != impl_fn.params.len) {
                    @compileError(std.fmt.comptimePrint("Parameter count mismatch in method '{s}' in type '{s}' required by '{s}'", .{ decl.name, @typeName(T), @typeName(I) }));
                }
                if (iface_fn.return_type != impl_fn.return_type) {
                    @compileError(std.fmt.comptimePrint("Return type mismatch in method '{s}' in type '{s}' required by '{s}'", .{ decl.name, @typeName(T), @typeName(I) }));
                }

                for (0..iface_fn.params.len) |i| {
                    if (iface_fn.params[i].type != impl_fn.params[i].type) {
                        @compileError(std.fmt.comptimePrint("Parameter type mismatch in method '{s}' in type '{s}' required by '{s}'", .{ decl.name, @typeName(T), @typeName(I) }));
                    }
                }
            }
        } else if (@TypeOf(member) == type) {
            if (!@hasDecl(T, decl.name)) {
                @compileError(std.fmt.comptimePrint("Expected type declaration '{s}' in type '{s}' required by '{s}'", .{ decl.name, @typeName(T), @typeName(I) }));
            }

            const impl_member = @field(T, decl.name);
            if (@TypeOf(impl_member) != type) {
                @compileError(std.fmt.comptimePrint("Declaration '{s}' in type '{s}' is required to be a type by '{s}', but got '{s}'", .{ decl.name, @typeName(T), @typeName(I), @typeName(@TypeOf(impl_member)) }));
            }
        } else {
            if (!@hasDecl(T, decl.name)) {
                @compileError(std.fmt.comptimePrint("Expected variable declaration '{s}' in type '{s}' required by '{s}'", .{ decl.name, @typeName(T), @typeName(I) }));
            }

            const impl_member = @field(T, decl.name);
            if (@TypeOf(impl_member) != @TypeOf(member)) {
                @compileError(std.fmt.comptimePrint("Declaration '{s}' in type '{s}' is required to be a variable of type '{s}' by '{s}', but got '{s}'", .{ decl.name, @typeName(T), @typeName(@TypeOf(member)), @typeName(I), @typeName(@TypeOf(impl_member)) }));
            }
        }
    }
}

This would then be used by declaring a "schema" and asserting against it:

const AnimalSchema = struct {
    pub const sound: []const u8 = "";
    pub fn pet() void {}
};

const Dog = struct {
    pub const sound: []const u8 = "bark";

    pub fn pet() void {
        // Do something
    }
};

const Cat = struct {
    pub const sound: []const u8 = "meow";

    pub fn pet() void {
        // Do something
    }
};

fn pet_animal(comptime T: type) void {
    assert_interface(AnimalSchema, T);
    T.pet();
}

pet_animal(Dog);
pet_animal(Cat);

Solaris Turnstiles

2026-09-08T00:00:00+00:00

Introduction

Sun microsystems' Solaris was once widely regarded as having some of the best symmetric multiprocessing (SMP) support among the operating systems of its time. Much of this technical strength came from innovations developed within the project.

Although Solaris is now mostly defunct ^[1], its influence remains substantial; technologies pioneered by Solaris can still be found across a wide range of software

A lot of Solaris' inventions have been described and talked about ad nauseam (such as the Slab Allocator), but one I rarely see discussed is its use of turnstiles.

Despite being relatively obscure, the idea has quietly spread far beyond Solaris. Variations of it can now be found in major operating systems, web browsers, and language runtimes. In fact, you’re probably using several implementations of the same basic concept right now!

Why turnstiles exist

Before going in depth on turnstiles, let's take a step back and figure out what problems Sun engineers were trying to solve.

One of the key design hallmarks of Solaris was that it made heavy use of blocking mutexes (or locks), in part because they could provide much better latency for high-priority tasks, which is important if you wanted something approaching soft real-time behavior.

The catch was that blocking mutexes came with problems of their own, two of which turnstiles were designed to address.

Mutexes are big

A blocking mutex needs more than just a single bit saying whether it’s locked. Somewhere, the kernel also needs to keep track of things like:

Who currently owns the lock;
which threads are waiting for it; and
The state needed to coordinate blocking and waking those threads.

You could store all of this directly in every mutex, but that gets expensive pretty quickly.

Highly scalable software tends to rely on fine-grained locking, i.e having lots of locks protecting relatively small pieces of state. The problem is that most of those locks will be uncontended most of the time, so dedicating a large amount of bookkeeping to every single one is mostly wasted space.

Keeping mutexes small makes fine-grained locking much cheaper. If adding another lock only costs a handful of bytes, developers can afford to use more of them instead of combining unrelated state behind a smaller number of coarse-grained locks.

Priority Inversion

Another significant problem that arises with the heavy use of blocking locks is priority inversion.

Typically, when a thread acquires a spinlock, it disables preemption (via mechanisms like spl, Irql or preempt_disable). Disabling preemption effectively puts the thread at the highest priority on the system, blocking any other task that might want to interrupt its work from doing so until the spinlock is released.

This is great for throughput, but latency can suffer if preemption stays disabled for too long.

Blocking locks, on the other hand, do not disable preemption, which allows for better latency behavior; high priority tasks can interrupt other lower priority tasks. This can also reduce throughput however, so there is no single best solution (though this can partly be worked around through the use of adaptive spinning)

Since threads keep their priority when holding a lock, an especially ugly situation can occur: What if high priority task A tries to acquire a lock currently held by low priority task B?

A blocks, waiting for B to release the lock. So far, so good. The problem is that B is still a low-priority task. Any medium-priority task that becomes runnable can preempt B, preventing it from making progress and releasing the lock.

In effect, our high-priority task is now stuck waiting behind work that should never have been able to delay it in the first place. This is called priority inversion, and in the worst case it can delay A for an unbounded amount of time.

So much for those latency guarantees!

Priority inheritance

Luckily, a bunch of smart people figured out a neat solution to this: priority inheritance.

In the same situation, B would temporarily inherit A's priority until it finishes its critical section, allowing it to run ahead of any medium-priority task that might otherwise get into its way.

At first glance, this seems pretty simple: when a high-priority thread blocks on a lock, boost the priority of whoever owns it. Things get more interesting, though, once locks start depending on other locks.

Consider the following scenario: A waits on B, but B itself is waiting on C. A could propagate its priority to B, but it would still have to wait for a potentially lower-priority C to release the lock. To correct this, A's priority needs to be propagated through the owner chain until it reaches C. This is called multi-hop priority inheritance.

What gets tricky is keeping track of these chains efficiently in the kernel.

Different operating systems have come up with different machinery for keeping track of these dependency chains. Turnstiles are the mechanism Solaris and a bunch of UNIX-derived operating systems use to do exactly that.

A few other notable approaches are:

AutoBoost on Windows; and
Linux’s rt-mutex priority-inheritance machinery.

Turnstiles

⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⣀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⢀⣴⣿⣿⣿⣿⣿⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⢀⣴⣿⣧⣤⣤⣤⣤⣼⣿⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⣿⡿⢋⣴⣿⠀⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⣶⡄⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠋⠰⠿⠟⠃⣀⣉⡉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠰⣦⡄⠉⠻⢿⣷⣦⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠀⢻⣿⡄⠀⠀⠈⠙⠻⣿⣶⣤⣀⠀⠀⠀⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⢿⣷⡀⠀⠀⠀⠀⠀⠙⠻⢿⣷⣦⣄⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠈⣿⣷⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠛⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠘⣿⣧⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠀⠸⣿⣇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠀⠀⠹⡿⠂⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⢸⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠈⠉⠉⠉⠉⠉⠉⠀⠀

At a high level, a turnstile is a data structure associated with a contended lock. It keeps track of the threads waiting on that lock, along with the information needed to propagate priority through the lock’s owner chain.

The clever part is that this state doesn’t have to live inside the lock itself.

Instead, each thread gets its own turnstile when it is created, in case it might contend on a blocking lock. This hinges on the fact that a thread can only be blocked on one lock at a time.

When a thread does block, it effectively donates its turnstile to the lock it is waiting on. That turnstile then becomes the place where the kernel keeps the lock’s waiters and priority-inheritance state.

This raises an obvious question: if the turnstile doesn’t live inside the lock, how does the kernel find it?

The answer is a hash table. The lock's address is hashed and is used to locate the turnstile currently associated with that lock. Keep in mind that the diagram below logically shows the hash table as having one turnstile per hash bucket, but in practice each bucket contains a chain of them: finding the appropriate turnstile then requires iterating over that bucket's chain to find the turnstile that points to the desired lock.

When a thread blocks on an already contended lock, it donates its turnstile onto a per-turnstile free list. When the lock is finally released, each thread simply grabs a turnstile from the freelist, it does not matter which; a turnstile is not bound to a specific thread.

Each thread keeps track of the turnstile it is currently blocked on, and each turnstile knows the current owner of the lock.

Turnstile diagram

Owner chains

To do multi-hop priority inheritance, the priority boosting code needs to follow each successive lock dependency. Conceptually, this amounts to walking a chain that looks like: turnstile->owner->turnstile->owner... until it reaches an owner that is not blocked on a synchronization object (this could be kept track by a waiting_on field in the thread structure).

The locking required to do this actually gets quite nasty, so I advise you read the comments in the Illumos implementation^[2] if you're hungry for more details. The gist of it is that there is a bucket lock -> thread lock locking hierarchy, and the successive bucket locks are trylocked. If the acquisition fails then everything is dropped and tried again as to avoid a deadlock between two CPUs concurrently doing a PI walk.

Limitations

Keen observers may have noticed a glaring limitation with this scheme: a turnstile has a single inheritor. So what about locks with multiple owners, such as reader-writer locks?

This is where things get awkward. If a writer is blocked behind several readers, there is no single owner for it to donate its priority to.

Instead, kernels that use turnstiles rely on heuristics or "good enough" solutions, such as picking the first thread that acquired the lock as the inheritor, or straight up giving up on priority inheritance for multi-owner locks altogether. Another possible solution is simply boosting a thread's priority to some given ceiling before taking a shared lock.

I believe that the only priority inheritance scheme that does support multiple inheritors is NT's AutoBoost, but its internals are not widely documented.

Influence

Many UNIX-derived operating systems started taking (copying) ideas from Solaris once they moved towards SMP support. As such, turnstiles are now found in many kernels of the same family.

Turnstiles are currently used in:

Solaris, where they originated;
Illumos;
macOS's kernel, XNU;
FreeBSD; and
NetBSD.

Additionally, some hobby OSes have adopted them:

Outside of operating systems

Interestingly, the same basic idea shows up outside of operating-system kernels as well. A few notable examples are:

WTF::ParkingLot is particularly interesting because it is widely used and looks remarkably similar to the space-saving half of turnstiles.

Just like a turnstile, a parking lot keeps the expensive waiter machinery outside of the lock itself. When a thread needs to block, the address of the lock is used to find the corresponding wait queue in a global table.

The important difference is that a parking lot is primarily concerned with putting threads to sleep and waking them back up; it does not provide the priority-inheritance machinery that made Solaris turnstiles special, as userspace does not need to concern itself with managing priority inheritance, and instead lets the kernel do it through mechanisms like PI-futex on Linux.

Resources

Turnstiles are well described in the following books:

The Design and Implementation of the FreeBSD Operating System; and
Solaris Internals

Additionally, they were described in a Solaris magazine here.

Footnotes

It survives to some extent through Illumos ↩
See turnstile_interlock(). ↩
I actually did manage to do multi-owner PI for a specific case of SMR read-sections, AFAIK this is novel! ↩

Linux qspinlocks

2026-03-17T00:00:00+00:00

Introduction

The Linux kernel has a variety of synchronization mechanisms, each suited to a different context and set of constraints. The simplest one of them all is the spinlock; spinlocks are used everywhere short, fast mutual exclusion is needed and blocking is either too costly or outright forbidden, such as in interrupt handlers and scheduler code. Today, Linux spinlocks are implemented through a mechanism called qspinlock. This article traces the evolution of spinlock designs that led to qspinlock, and presents a sample userspace C++ implementation along with benchmarks.

Spinlocks

Spinlocks are the simplest synchronization primitive to implement; as their name indicates, they spin on the lock until it is released.

Artist's impression of a CPU spinning on a lock

Here is a sample implementation using the Test-and-Test-And-Set (TTAS) pattern, which is generally considered to be the most efficient:

struct SpinLock {
  std::atomic<uint32_t> v{0};
  void lock() {
    for (;;) {
      uint32_t expected = 0;
      if (v.compare_exchange_weak(expected, 1, std::memory_order_acquire,
                                  std::memory_order_relaxed))
        return;
      while (v.load(std::memory_order_relaxed) != 0)
        cpu_relax();
    }
  }
  void unlock() { v.store(0, std::memory_order_release); }
};

It is important to note that spinlocks are inherently unfair: when the lock is released, all waiting CPUs race for it simultaneously, and there is no guarantee that the longest-waiting CPU will win. A CPU that is topologically closer to the releasing CPU, such as one sharing an L3 cache or sitting on the same NUMA node, will tend to win repeatedly, potentially starving remote CPUs indefinitely under sustained contention.

Ticket spinlocks

One way to implement fairness in spinlocks is using Ticket spinlocks, such locks work similarly to a deli counter: each waiter draws a ticket and is served strictly in order, providing a First-In-First-Out (FIFO) ordering.

These locks have the advantage that they provide a good bang for the buck: they are simple to implement, performant and provide guaranteed fairness.

Here is a sample implementation:

struct TicketLock {
  std::atomic<uint32_t> next{0};
  std::atomic<uint32_t> owner{0};
  void lock() {
    uint32_t my = next.fetch_add(1, std::memory_order_relaxed);
    while (owner.load(std::memory_order_acquire) != my)
      cpu_relax();
  }
  void unlock() { owner.fetch_add(1, std::memory_order_release); }
};

Before moving to qspinlock, Linux used something similar for its locks, albeit fitting them in a single 32-bit value.

qspinlock

Motivations

If ticket spinlocks are so great, why did Linux switch away from them? The answer to this question lies in scalability: when a large number of cores contend on the same lock, this causes cache line bouncing, where the cache line of the lock is repeatedly transferred between CPUs, and each unlock invalidates every waiting CPU's cached copy simultaneously. With N waiters, a single unlock triggers N cache line transfers, increasing coherence traffic and causing latency to degrade as contention grows.

MCS Locks

Note: MCS Locks are better explained in this LWN Article and most of the information presented here comes from it.

One mechanism that solves this is the MCS lock (named after its authors), which eliminates cache line bouncing by giving each waiter its own cache line to spin on.

An MCS lock (or node) can be defined as such:

struct McsNode {
  std::atomic<McsNode *> next;
  std::atomic<int> locked; // 1 when acquired
};

The way MCS locks work is via a sort of linked list, where each new waiter atomically appends its node to the tail of the main lock (called next), then spins on its own locked field until its predecessor wakes it up.

Let us walk through the process of acquiring and releasing an MCS lock.

First, CPU0 atomically swaps the tail pointer with a pointer to its own node and notices the previous value was NULL, indicating the queue was empty and the lock is now held by CPU0:

Uncontended case

Now, CPU1 tries to acquire the lock, but it is contended: it notices the return value of the atomic exchange is non-NULL and sets the old tail's next value to its own McsNode, and spins on its own locked value until it is 1, ensuring spinning on CPU-local data:

Contended case

When CPU0 releases the lock, it attempts to compare-and-swap (CAS) the tail pointer back to NULL. If the CAS succeeds, the queue is empty and the lock is free. If it fails, a successor has arrived, CPU0 then sets CPU1->locked = 1, waking CPU1 and handing off the lock.

CPU1 has successfully taken the lock

Mechanism

At their core, qspinlocks are based on MCS locks, but they do not have MCS nodes directly embedded within them, as a means to save space and keep the lock to a single 32-bit word. That word is structured as such:

Layout of a qspinlock

In this implementation, we will define a qspinlock as:

struct QSpinLock {
  union {
    std::atomic<uint32_t> val;

    struct {
      std::atomic<uint8_t> locked;
      std::atomic<uint8_t> pending;
    } low_word;

    struct {
      std::atomic<uint16_t> locked_pending;
      std::atomic<uint16_t> tail;
    } split;
  };
};

Here be dragons 🐉! Accessing multiple fields of a union is undefined behavior (UB) in C++. However, in practice all major compilers (GCC, Clang) treat this as well-defined. We accept the UB and move on.

Let's go through every field one by one:

locked: Set to 1 when the lock is held, and cleared on release. Waiters at the head of the queue spin on this byte waiting for it to reach 0 before taking the lock.
pending: Used as an optimization when there is only one CPU waiting for the lock to be freed. Rather than setting up a full MCS queue node, the first waiter simply sets this bit and spins on locked.
tail: Split into two subfields: the lowest two bits encode the qnode slot index and the remaining bits encode the CPU ID of the tail node.

qnodes

The value encoded in the tail field indexes into a per-CPU array of MCS nodes called qnodes. Linux allocates 4 nodes per CPU. The reason there is an array rather than a single node is to handle nested acquisition: if an interrupt fires on a CPU that is already waiting on the lock, it needs a separate node, so each CPU maintains a small array of nodes instead of just one (though it is discouraged to take locks in interrupts anyway!).

Let's now go through the implementation of locking.

Locking

Note: This implementation is mostly based on the actual Linux code, adapted for C++ and omitting extraneous checks

Constants

Let us first define a few useful constants related to extracting fields from the 32-bit word:

  static constexpr uint32_t LOCKED_MASK = 0x000000FF;
  static constexpr uint32_t LOCKED_VAL = 0x00000001;

  static constexpr uint32_t PENDING_MASK = 0x0000FF00;
  static constexpr uint32_t PENDING_VAL = 0x00000100;

  static constexpr uint32_t TAIL_IDX_MASK = 0x00030000;
  static constexpr uint32_t TAIL_CPU_MASK = 0xFFFC0000;
  static constexpr uint32_t TAIL_MASK = TAIL_IDX_MASK | TAIL_CPU_MASK;

  static constexpr uint32_t TAIL_IDX_OFFSET = 16;
  static constexpr uint32_t TAIL_CPU_OFFSET = 18;

Each _MASK constant isolates its field via a bitwise AND, for example, val & LOCKED_MASK extracts the locked byte. Each _VAL constant is the value written to set that field, LOCKED_VAL is 1 in bit 0, PENDING_VAL is 1 in bit 8. The _OFFSET constants are the shift amounts used when encoding and decoding the tail field: the slot index sits at bit 16, and the CPU ID at bit 18.

The fastest path

The fastest path happens when we try to grab an uncontended lock. In that case, we can just set the locked bit and be done:

void lock() {
    uint32_t v = 0;

    // Fastest path
    if (val.compare_exchange_strong(v, LOCKED_VAL, std::memory_order_acquire,
                                    std::memory_order_relaxed)) {
      return;
    }

    // ...
  }

The medium path

The medium path happens when we try to grab a contended lock with no waiter. First, we check whether the only bit set is pending, this indicates that a pending waiter is mid-transition, in the process of atomically clearing pending and setting locked to take ownership. We spin for up to PENDING_LOOPS iterations to let that transition complete:

if (v == PENDING_VAL) {
    uint32_t cnt = PENDING_LOOPS; // 512 on Linux
    while (cnt--) {
        v = val.load(std::memory_order_relaxed);
        if (v != PENDING_VAL)
            break;
        cpu_relax();
    }
}

Then, we check whether any bit but locked is set (i.e. either tail or pending is non-zero), which would indicate another waiter, in that case we go to the slow path:

if (v & ~LOCKED_MASK) {
    queue(); // Go to slow path
    return;
}

As far as we know, there is no one contending on the lock, so we can set the pending bit:

v = val.fetch_or(PENDING_VAL, std::memory_order_acquire);

However, fetch_or is not a CAS, another CPU may have raced us and modified the lock word between our initial check and this store. So we validate the value returned by fetch_or and check for contention once more:

if (v & ~LOCKED_MASK) [[unlikely]] {
    if ((v & PENDING_MASK) == 0) {
        // We set the pending bit but we shouldn't have, undo it.
        val.fetch_and(~PENDING_VAL, std::memory_order_release);
    }
    queue(); // Go to slow path
    return;
}

Now, if the lock is still held, we spin on locked until it is cleared by the current holder:

if (v & LOCKED_MASK) {
    while (low_word.locked.load(std::memory_order_acquire) != 0) {
        cpu_relax();
    }
}

Once locked is clear, we atomically clear pending and set locked in a single 16-bit store:

split.locked_pending.store(1, std::memory_order_release);

The slow path

The slow path occurs when we have no choice but to queue on the lock. We first attempt to allocate an McsNode from our per-CPU stash by claiming a slot index from curr_idx, if there is none available we have no choice but to spin on locked:

void queue() {
    uint32_t idx = my_cpu->curr_idx++;

    struct IdxGuard {
      int &v;
      ~IdxGuard() { v--; }
    } guard{my_cpu->curr_idx};

    if (idx >= MAX_IDX) {
      while (!try_lock())
        cpu_relax();
      return;
    }

  // ...
}

The IdxGuard RAII pattern ensures the slot is released when queue() returns, regardless of which path we exit through. If all slots are exhausted, which should very rarely happen in practice, we fall back to spinning directly on try_lock() (which simply tries to CAS val with LOCKED_VAL).

We then initialize the node and make one last opportunistic attempt to grab the lock. As the Linux source notes, we have just touched a (possibly cold) cache line to initialize the node and someone may have released the lock while we weren't watching, so it is worth trying once more before committing to the queue:

McsNode *node = &cpus[my_cpu->id]->qnodes[idx];

node->locked.store(0, std::memory_order_relaxed);
node->next.store(nullptr, std::memory_order_relaxed);

// Try one last time
if (try_lock()) {
    return;
}

Now we have absolutely no choice but to queue, first encode the tail index:

uint32_t tail = encode_tail(my_cpu->id, idx);

Where encode_tail is:

uint32_t encode_tail(uint32_t cpu_id, uint32_t idx) {
    return ((cpu_id + 1) << TAIL_CPU_OFFSET) | (idx << TAIL_IDX_OFFSET);
}

Note the cpu_id + 1: CPU IDs start at 0, so an ID of 0 would be indistinguishable from an empty tail field. Adding 1 ensures that a non-zero tail always indicates a real waiter.

We then atomically exchange the tail field with our new value, retrieving the previous tail:

uint32_t old = xchg_tail(tail);

Where xchg_tail operates only on the upper 16 bits of the lock word, leaving locked and pending untouched:

uint32_t xchg_tail(uint32_t tail) {
    uint16_t tail16 = tail >> 16;
    uint16_t old_tail16 =
        split.tail.exchange(tail16, std::memory_order_acq_rel);
    return static_cast<uint32_t>(old_tail16) << 16;
}

Now, we need to check whether or not we have a predecessor, and if so, we have to let it know of our presence by setting its next field to point to our node. We then enter phase 1 of the two-phase wait, spinning on our own node->locked until our predecessor wakes us:

// Someone else was already waiting, set their next
if (old & TAIL_MASK) {
    McsNode *prev = decode_tail(old);

    prev->next.store(node, std::memory_order_release);

    while (node->locked.load(std::memory_order_acquire) == 0) {
        cpu_relax();
    }
}

The release store to prev->next and the acquire load of node->locked ensures the predecessor will not write to node->locked until it has observed our node via its own next pointer.

decode_tail simply undoes the encoding to recover the CPU ID and slot index:

McsNode *decode_tail(uint32_t tail) {
    uint32_t cpu_id = (tail >> TAIL_CPU_OFFSET) - 1;
    uint32_t idx = (tail & TAIL_IDX_MASK) >> TAIL_IDX_OFFSET;
    return &cpus[cpu_id]->qnodes[idx];
}

Once our predecessor has woken us, we are now the head of the queue. We enter phase 2: spinning on the main lock word until both locked and pending are clear, meaning the current holder has released the lock and no pending waiter is mid-transition:

uint32_t v = 0;

while (((v = val.load(std::memory_order_acquire)) &
        (LOCKED_MASK | PENDING_MASK)) != 0) {
    cpu_relax();
}

Now we attempt to take ownership. If we are the only waiter in the queue, meaning the tail still points to our node, we try to atomically claim the lock and clear the tail in one CAS, leaving the lock word clean:

if ((v & TAIL_MASK) == tail) {
    uint32_t expected = v;
    if (val.compare_exchange_strong(expected, LOCKED_VAL,
                                      std::memory_order_acquire,
                                      std::memory_order_relaxed)) {
        // No contention, we're done
        return;
    }
}

If the CAS fails, a new waiter has joined the queue behind us since we last checked. In that case we cannot clear the tail, so we take a different approach: we set locked directly to signal ownership, then promote our successor to queue head by setting their node->locked to 1:

low_word.locked.store(1, std::memory_order_release);

McsNode *next = nullptr;
while ((next = node->next.load(std::memory_order_acquire)) == nullptr) {
    cpu_relax();
}

next->locked.store(1, std::memory_order_release);

The loop on node->next handles the inherent race in the two-step enqueue: a successor may have already swapped the tail but not yet written their next pointer. We simply wait for them to finish. Once next is visible, a single store to next->locked wakes the successor and hands off the queue head, completing the acquisition.

Summary

To summarize, the lock() function has three paths of increasing complexity:

Fast path: the lock is uncontended, a single CAS is sufficient.
Medium path: the lock is held but the queue is empty, we set the pending bit and spin on locked without touching the MCS queue.
Slow path: the queue is occupied, we enqueue an MCS node, wait for our predecessor to promote us to head, then spin on the main lock word before taking ownership.

Unlocking

Phew, that was a lot to uncover for locking! Thankfully unlock() is very simple:

void unlock() { 
    low_word.locked.store(0, std::memory_order_release);
}

This is actually the reason why we use a whole byte for locked instead of a bit: unlock() is as simple as an atomic byte store!

Benchmarks

It's fun to nerd out on implementation details but how much faster is qspinlock?

I have set up a simple benchmark, measuring SpinLock against TicketLock and QSpinLock, it consists of creating N threads (up to nproc) all pinned to their own CPU cores taking the lock and pushing 500 000 elements to a std::vector<int>:

for (int j = 0; j < OPS_PER_THREAD; j++) {
    lock.lock();
    shared_vector.push_back(j);
    lock.unlock();
}

We then wait for each thread to complete and measure the time taken:

  auto t0 = std::chrono::steady_clock::now();
  start.store(true, std::memory_order_release);
  for (auto &t : threads)
    t.join();
  auto t1 = std::chrono::steady_clock::now();

Here are the results:

Benchmark executed on a Ryzen 5 5600X

Threads	Spinlock (ops/s)	Ticket Spinlock (ops/s)	qspinlock (ops/s)
1	551,081,387	283,366,874	159,177,979
2	31,101,649	15,260,200	16,609,980
4	14,855,526	7,864,148	16,271,328
8	9,575,916	4,925,312	16,911,329
12	5,384,959	3,107,082	13,454,086

We can see the naive spinlock is much faster uncontended, which is expected since it is the simplest (albeit unfair). We can also see qspinlock seems to scale better than both the naive spinlock and the ticket spinlock, maintaining relatively stable throughput as thread count increases.

Why is it so slow uncontended though? In theory, qspinlock should be as fast as a spinlock in the uncontended case, so why is it not? Honestly, I'm not sure, but my best guess is that it is because of the very short critical section causing a store-to-load forwarding stall: since we write to locked as a byte in unlock(), but access the value as a 32-bit integer in lock(), the CPU pipeline stalls and we end up with poor performance. In fact, changing unlock() to:

void unlock() {
    val.fetch_and(~LOCKED_VAL, std::memory_order_release);
}

Yields the same uncontended performance as the ticket lock, which is expected since both need a Read-Modify-Write (RMW) operation. This is slower overall, though. Changing this to a simple store over the whole value (which is actually incorrect), yields the same uncontended performance as spinlock.

In practice, real critical sections are rarely that short, the additional work done while holding the lock gives the pipeline enough time to resolve the stall, so this might not be an actual concern. (Also, it only happens uncontended!)

Summary

qspinlock is not a silver bullet and its implementation is significantly more complex than either a plain spinlock or a ticket lock. However, on a machine with many cores, as many as Linux needs to run on, the CPU-local spinning does make a measurable difference under sustained contention, even on my puny 6c/12t machine.

If you wish to dig deeper, I highly recommend:

Reading the Linux code
Reading this LWN Article

The code written for this article is available here

Thank you for making it this far!