Projects / Weir — Durable Write Buffer

Weir — Durable Write Buffer

activeRust~345 µs · durable ack p50

Production-deployed durable write-ahead buffer in Rust (v1.3.1). Fire a record at a local Unix socket, get a durable fsync'd ack in microseconds, then let the daemon drain records to your database in batches — turning N per-record commits into one server-side commit. An acked record is on disk and replays after a crash; the daemon never returns a false ack. v1 wire protocol and Rust API frozen under SemVer.

STATUSactive
LANGUAGERust
METRIC~345 µs · durable ack p50
STACKRust · Write-Ahead Log · Durability · Storage Systems · MySQL · Postgres · ClickHouse · mTLS · Unix Sockets
COMMITS601
STARS0
601 commits‹› Code ↗
M
miki-przygodachore: update benchmark baselines [skip ci]0b8151b5 days ago
·.dockerignore
·.gitattributes
.github
·.gitignore
·CHANGELOG.md
·CONTRIBUTING.md
·Cargo.lock
·Cargo.toml
·LICENSE
·README.md
·SECURITY.md
·book.toml
crates
demo
demos
·deny.toml
deploy
docs
fuzz
scripts
📄
weir

A durable, high-throughput write buffer for Rust.

**Fire a record at a local socket, get a durable ack in microseconds, and let

your database catch up in bulk.** weir writes each record to a CRC32-checksummed

write-ahead buffer, fsyncs it according to the durability tier you ask for, acks

the producer, then drains records to your sink in batches — turning N per-record

commits into 1. An ack is never a false ack: an acked record is on disk and

~69 µs Buffered (non-durable) ack p50 · ~364 µs durable Sync ack p50 ·

~2,550 Sync RPS · ~58,600 RPS at saturation (Buffered, ~64 threads)

(indicative — single box, sandboxed CI runners; reproduce on your own hardware, see benchmarks) ·

5 built-in sinks (4 in a default build; clickhouse opt-in) · v1 wire + Rust API frozen under SemVer

▶ Try the demo — a self-contained, browser-only

simulation of the pipeline: push records, flip the durability tier, crash the

daemon mid-flight, and watch unconfirmed segments replay, side by side with a

naive insert-per-record baseline. No build step — open demo/index.html in any

browser. (Hosted version coming with the public launch.)

> Status — 1.x (stable). The v1 wire protocol and public Rust API (weir-core,

> weir-client, weir-sink-sdk, weir-wab) are frozen under

> language-neutral conformance suite pinning the wire

> format for non-Rust implementers. The WAB on-disk format is stable and

> unconfirmed segments replay on restart. Built-in sinks: noop, http,

ABOUT

A durable, high-throughput write buffer for Rust - microsecond fsync'd acks over a Unix socket, N→1 batched commits to your database, and crash-safe WAB replay.

batchingclickhousedaemondurabilityfsyncprometheustokiounix-socketwrite-ahead-loggingwrite-buffer
ACTIVITY
Stars0
Forks0
Commits601
LicenseMIT
LANGUAGES
Rust94.7%
Shell2.5%
Python2.4%
Dockerfile0.4%
HOW IT WORKS
  • Producers write records over a Unix domain socket (or TCP + mutual TLS); the daemon validates a dual-CRC32 frame (header + payload), appends to the write-ahead buffer (WAB), group-fdatasyncs at the batch boundary, ACKs, then asynchronously drains batches to a pluggable sink.
  • Three durability tiers: Sync and Batched are durable-before-ack (fdatasync before the ACK); Buffered acks on enqueue (~72 µs p50), trading power-loss safety for latency.
  • WAB segments aren't reclaimed until the sink confirms the batch — on restart, unconfirmed segments replay automatically. An ack is never a false ack.
  • Durable Sync-tier ack p50 ~345 µs; ~67.7k records/sec (Buffered) and ~33.8k (Sync) at saturation — fsync-bound (single CI box; reproduce on your own hardware).
  • Deployed to production at bjj.space.
ARCHITECTURE
  • Seven-crate workspace: weir-core (dependency-light wire types — crc32fast + bytes, no serde), weir-wab (on-disk segment format + reader, shared with the CLI), weir-server (Unix-only daemon), weir-client, weir-sink-sdk (custom-sink trait), weir-ctl (admin CLI), weir-testkit.
  • Five built-in sinks: noop (soak-testing), http (NDJSON, transient/permanent error classification), mysql + postgres (N records → one multi-row INSERT → one server-side commit; Postgres ON CONFLICT dedupe), plus clickhouse (RowBinary, SHA-256 dedup token) behind an opt-in Cargo feature.
  • Per-shard MPMC queue for concurrent producers without a global lock; per-shard draining with partial-batch success and a dead-letter quarantine that pauses — never drops — when full.
  • WAB flusher and drain threads are panic-supervised — catch_unwind observes a panic, increments a metric, and respawns up to 10× before taking the shard offline; a panic never produces a false ack.
  • Default transport is a 0o600 bind-hardened Unix socket with peer-uid checks (SO_PEERCRED / getpeereid); optional TCP + mutual TLS (CA-signed client certs, handshake timeout) behind the tls feature.
CORRECTNESS & TESTING
  • Deterministic Simulation Testing: a seeded fault engine injects failed fsyncs, torn writes, crashes between sync and rename, and mid-flush panics — 300 seeds × 5 scenarios (~1,500 fault runs) every CI push, proving 'acked ⇒ durable' with proptest auto-shrinking to a minimal reproducer.
  • v1 wire protocol pinned by 30 language-neutral conformance vectors covering every message type and all 9 Nack reasons — so non-Rust clients (Python, Go, C, Java, TS) can verify their own codec.
  • 46 system tests (SIGKILL-and-restart crash recovery → replay → drain → confirm), plus 2 cargo-fuzz targets on the envelope and WAB parsers.
  • 30+ Prometheus metric families with a turnkey docker-compose monitoring stack and Grafana dashboards; SemVer-frozen v1 API and on-disk format.