In my
first Squirix article,
I wrote about why Squirix keeps a strict boundary between the client and the server. Applications use a typed client
over gRPC; the server owns cache state, routing, persistence, recovery, and operational endpoints.
That boundary makes ownership clear. It does not answer the uncomfortable question underneath it: what exactly happens
between accepting a write and telling the client it succeeded?
Preview.6 goes one level deeper into that path. What if the process dies after writing to disk but before sending the
response? What survives a torn journal tail? How can a client retry without applying the same mutation twice?
Those questions shaped the release: a pipelined binary write-ahead log, tighter snapshot and journal recovery, and
idempotent mutation outcomes that survive a restart.
Squirix is still an experimental preview, not a production-ready cache. Its APIs and storage formats may change during
0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.
Follow one write
The basic ordering rule is simple:
Append the mutation to the journal before applying it in memory, and do not return success before the required
durability boundary.
For a normal durable mutation, the path looks like this:
validate and admit the mutation
|
v
append a binary WAL record
|
v
cross the configured durability boundary
|
v
apply the mutation in memory
|
v
return success
The order is deliberate. If the server changed memory first and appended to the journal afterward, a crash in between
could make an acknowledged write disappear after restart.
Idempotent RPCs take one deliberate variation. Squirix appends the mutation, applies it in memory to produce the
response, appends that response as an idempotency outcome, then waits for a single durability boundary covering both
records. The response is not returned before that wait completes.
The mutation and its retry result share one durability commit, without pretending the network response is atomic with
the disk write.
Journal-first ordering avoids acknowledged-but-unrecoverable writes, but it exposes a different case. Suppose the
journal becomes durable and the process dies before the response reaches the client. Recovery may replay the mutation
even though the client observed only a timeout or broken connection.
That outcome is not an ordinary failure. It is commit unknown: the operation may have committed, but the client did
not receive a definitive answer.
This is different from a rejection before the journal. If the memory-admission gate rejects a growing entry, Squirix
has not appended anything and has not changed cache state. That rejection is definitive. Once durable bytes may exist,
the answer has to be more careful.
Moving file I/O off request threads
Preview.6 replaces the previous journal path with a pipelined binary WAL backend. The goal was not simply to make file
writes faster. It was to give journal work one owner and make backpressure visible.
Request threads do not independently open files, encode records, and call fsync. They submit append work to a bounded
ring. A dedicated background thread named squirix-journal-io owns the journal event loop:
request threads
| | |
v v v
+-------------------+
| bounded WAL ring |
+-------------------+
|
v
+------------------------+
| single journal thread |
+------------------------+
|
v
binary WAL segments
The journal thread drains accepted work, encodes and batches frames, writes segment bytes, services durability
deadlines, performs flushes, and coordinates segment rolls.
The queue is bounded because storage can always fall behind incoming traffic. An unbounded queue would turn that
mismatch into unbounded memory growth; the ring turns it into visible backpressure instead.
The single writer also gives segment state one clear owner. Offsets, batches, durability state, and roll decisions do
not need to be coordinated among arbitrary request threads.
Why the journal is binary
The new WAL writes binary frames rather than passing persistence records through JSON text.
That gives the storage path predictable record sizes, explicit field validation, CRC32C checksums, and span-based
encoding for the hot parts of the format. It also makes versioning a property of the storage format rather than an
accidental consequence of a general-purpose serializer.
This is partly a performance choice, but the main benefit is control. Recovery needs to distinguish a complete record
from arbitrary bytes at the end of a file. A framed binary format can say exactly how long a record is and whether its
checksum is valid.
The event loop also coalesces encoded records into write batches. Writing bytes and declaring those bytes durable are
separate steps, which is where group commit enters the design.
Sharing an fsync with group commit
Flushing every mutation separately gives straightforward semantics, but it also makes each request pay the full
durability cost.
With group commit enabled, several appends can share one flush:
append A ─┐
append B ─┼── write batch ── fsync ── complete A, B, and C
append C ─┘
Each waiter is completed only after a durability flush covers the relevant appends. The journal thread flushes the
current write batch, performs the storage flush, and then completes the captured batch of waiters. If the flush fails,
every waiter in that batch fails; no caller is told that partial durability was enough.
The group is bounded by configuration—a maximum wait and a maximum batch size—trading a small, controlled delay for
fewer durability flushes. Workloads that prefer the strictest per-write latency can disable group commit.
The important contract is not merely that an fsync happened somewhere. It is that a completed waiter is covered by
that durability flush.
The response can still get lost
Even with correct WAL ordering, the server cannot make the network response atomic with the disk write.
The failure window is easiest to see as a short sequence:
1. append operation A
2. flush operation A
3. crash before the response reaches the client
4. recover operation A from the WAL
5. client retries operation A
Without a stable request identity, step 5 could apply the same logical mutation twice. The risk is greatest for
mutations whose effect is not naturally idempotent.
Squirix mutation RPCs therefore carry an opaque operation_id. The server associates the ID with a request fingerprint
and the serialized response. A retry with the same ID and fingerprint can return the original result. Reusing the ID
for a different request is rejected.
In preview.6, that identity survives restart. Recovery rebuilds successful durable outcomes from operation metadata in
the journal and from retained idempotency records in snapshots. The operation ID is also preserved when a receiving
node forwards a mutation to the key owner.
client
|
| operation_id = 4f...
v
receiving node
|
| same operation_id
v
owner node
|
+--> durable mutation record
|
+--> durable outcome metadata
This is not a distributed transaction, and I will not present it as one. It gives a supported mutation a stable identity
across forwarding, timeouts, retries, and restarts—exactly the situations where commit unknown becomes operationally
important.
Recovery starts by distrusting the files
A snapshot is not valid merely because a file with the right name exists. Squirix reads and validates the complete
snapshot into a temporary load result before applying its entries. The startup gate remains closed until snapshot
restore and journal replay have both completed.
The current recovery flow is:
- Read the manifest.
- Locate the referenced snapshot.
- Decode cache entries and retained idempotency records into a temporary load result.
- Validate the complete snapshot.
- Apply the validated snapshot state.
- Replay journal records after the snapshot watermark.
- Open the startup gate.
manifest
|
v
snapshot ── validate all frames ── apply recovered state
|
v
last applied sequence N
|
v
replay WAL records where sequence > N
The watermark matters. A snapshot represents state through a particular journal sequence. Replaying records at or
before that sequence would apply mutations twice; skipping later records would lose committed state.
Preview.6 freezes the metadata that describes the snapshot cut so that the entries and their sequence boundary refer
to the same logical point.
Snapshot publication follows the usual temporary-file pattern: write and flush a temporary file, publish the completed
snapshot, and only then update the manifest reference. A crash during the temporary write must not make an incomplete
snapshot authoritative.
If the referenced snapshot is missing, unreadable, truncated, or fails checksum validation, Squirix discards all
snapshot-derived state. It falls back to journal-only recovery only when the required journal history is still
available. Otherwise recovery fails instead of guessing across a gap. Preview.6 also fixes the valid fallback path so
replay starts from the earliest available segment when there is no snapshot watermark.
A torn tail does not erase the valid prefix
A process can stop halfway through the final WAL frame. Recovery must not interpret those bytes as a complete mutation,
but it should not discard every earlier record either.
[valid frame][valid frame][valid frame][partial bytes...]
^
recovery stops here
Journal replay proceeds through complete, validated records in segment order. During startup, Squirix scans the active
segment to the last valid frame boundary and truncates an incomplete tail before reopening it for writes.
This is intentionally conservative:
- never invent a record from partial bytes;
- never replay beyond an invalid boundary;
- preserve the valid committed prefix.
The same principle applies to snapshots: do not start applying entries until the complete file has been decoded and
validated. A clean journal fallback is easier to reason about than trusting a partly decoded snapshot.
Compaction must preserve retry semantics
Compaction cannot keep only the latest key/value state.
Imagine that operation A committed, its result was retained for deduplication, and old journal segments were then
compacted away. If the snapshot contained cache values but not the idempotency record, a delayed retry of A could lose
its original identity and execute again.
Snapshots therefore include retained idempotency outcomes alongside cache entries. After compaction and restart,
Squirix needs to recover both:
cache state
+
retry identity and durable outcomes
Durability is not only the ability to reconstruct values. It also means reconstructing enough protocol state to handle
uncertain retries safely.
What preview.6 changes
The durability work in 0.1.0-preview.6 includes:
- a pipelined binary WAL;
- a dedicated single-writer journal event loop;
- bounded append backpressure;
- configurable group commit;
- journal-only recovery from the earliest available segment when the journal topology is valid;
- a stable snapshot cut and replay watermark;
- durable operation outcomes across restart;
- operation-ID propagation between nodes;
- retained idempotency records in snapshots;
- compaction cancellation during shutdown;
- lower allocation pressure on WAL hot paths.
Some changes improve throughput. Most exist because of awkward timing: a crash after a journal flush, a lost response,
a truncated final frame, compaction followed by an old retry, or shutdown while persistence work is active.
That is where durability engineering lives. The happy path is necessary, but it is rarely the part that keeps me
thinking after the code is written.
What preview.6 does not promise
This work makes the durability model easier to reason about, but Squirix is still early software.
The on-disk format may change during 0.x. Durability is per node; preview.6 does not add replication or automatic
failover. Operations spanning multiple owners are not globally atomic transactions. Serializer compatibility and
restart behavior still need to be validated for the mutation paths a workload depends on.
A WAL is not a substitute for replication, backups, or recovery testing. Preview.6 is a stronger durability foundation,
not a production-readiness declaration.
Closing thought
Appending bytes is the easy part of a write-ahead log. Deciding what those bytes mean after a crash is the real work.
The difficult part is naming every boundary precisely:
- accepted;
- appended;
- flushed;
- applied in memory;
- visible to the client;
- recoverable after restart;
- safe to retry.
Preview.6 makes those boundaries more explicit. That gives future work—replication, failover, and more advanced cluster
behavior—a durability model it can build on instead of work around.
If you work on storage engines or distributed .NET systems, I would be glad to hear where you think the remaining sharp
edges are. The project is open source, and feedback on the WAL, recovery semantics, and retry model is especially
welcome.
