RailPredict -- UK Rail Engine

activeRustlive · Darwin firehose

Real-time UK rail data engine in Rust (v1.21.2, MIT-licensed). Subscribes directly to National Rail's Darwin Push Port STOMP firehose, tracks every train in a concurrent in-memory registry, and serves every query from local state -- the rate-limited GBR API is never on the request path. LightGBM delay models run in-process via ONNX; their published accuracy was withdrawn after a leakage review and a leak-free retrain is in progress.

▶ Try the demo GitHub ↗

STATUSactive

LANGUAGERust

METRIClive · Darwin firehose

STACKRust · Darwin Push Port · STOMP · Postgres · sqlx · axum · htmx · maud · LightGBM · ONNX

REPOmiki-przygoda/RailPredict ↗

ARCHITECTURE

Zero-poll ingestion -- a custom tokio STOMP/TLS client subscribes to the Darwin Push Port firehose (every UK train movement); messages are parsed with quick-xml and filtered by RID into a concurrent DashMap registry that serves read-only queries. Unwanted message types are rejected from raw bytes before any XML parsing, a per-train timestamp guard drops replays, and field updates merge by version so a stale source never overwrites live data.
Three-tier data model: Tier A (GTFS static timetables, days–weeks staleness), Tier B (local statistical + ML delay prediction, minutes–hours) and Tier C (live GBR Retail API, reserved for final ticket purchase and not yet wired). The vast majority of queries are answered from Tiers A and B alone.
Five urgency states per train (Dormant → Monitored → Active → Critical → Terminal), set inline by ingestion; cancellations and delays promote a train to Critical and drive the SSE stream, and high wind (Open-Meteo) flags volatility.
Built, not wired: a request coalescer (many watchers of one train collapse to one outbound call, fanned back via oneshot channels), a token bucket and a typed circuit breaker for the GBR API. They are tested but not on a live path, because Tier C is not connected yet.
axum + maud server-rendered HTML with htmx (no JS framework, no build step); SSE live stream at /trains/:rid/live; Prometheus metrics and tower_governor rate limiting (60 req/s per IP). /health returns 503 when the Darwin feed goes quiet, with a darwin_feed_lag_seconds gauge.

PREDICTION & ML

Two-model LightGBM cascade exported to ONNX and run inside the Rust binary: a day-ahead model (no live signal) and a real-time model that folds in the train's current reported delay, with a trimmed-mean statistical fallback when the models are unavailable.
Accuracy withdrawn (v1.21.1): the earlier published MAEs came from a training feature derived from the label, a random split of time-series data, and labels from before a delay-computation fix. They are no longer quoted.
Training pipeline fixed: the real-time target is a journey's final delay predicted from an earlier snapshot's own reading, splits are by date, rolling stats use prior dates only and encodings are fitted on training dates only, with DB-free leakage tests.
The first honest retrain, on five days of post-fix data, lost to trivial baselines (persistence and a per-service mean), so no new model is shipped yet; that result is documented rather than tuned away.

ENGINEERING & RIGOR

Every HTML template is type-checked Rust (maud), so template mistakes are build errors; DB queries are exercised by sqlx integration tests against a real Postgres.
340+ tests (unit tests across the modules plus sqlx DB and HTTP integration suites over a Postgres service container); CI runs cargo-deny (licence + vulnerability), clippy -D warnings, the test suite and a release build as parallel jobs.
Full-journey capture -- per-stop arrival/recovery deltas accumulated across partial STOMP frames and persisted on deactivation; sticky cancellation detection; RDS reference data seeds station and operator names (81 operators), and 2,751 GB stations are geolocated for the live map.
Live /map of running trains snapped to the GB rail graph, plus RailPredict OS -- a self-contained, fully offline desktop-style demo (Live Map, Operators, Predictions, Reliability, Replay) that bakes real historical data into a single HTML file.