Performance
On a 2 vCPU / 8 GB cloud VM shared with PostgreSQL, Kafka and the load driver, Tandem delivers COMMIT → ack at a median of 13 ms and a p99 of 87 ms while carrying 600 events a second, and sustains 1200 to 1400 events/s.
Zero lost events and zero ordering violations in every scenario, at every rate — including the rates the host could not keep up with.
COMMIT → ack at 600 events/s, mean of three runs. p95 46 ms, p99 87 ms; the p99.9 is the host’s own tail and ranged 146 to 363 ms.
Highest rate delivered with the backlog flat, across seven searches in one hour on two cores shared with the database and the broker.
Across all seven scenarios — saturation past the ceiling, a relay killed mid-flight, an undeliverable event, three instances with one killed outright.
Latency
The clock starts when the producing transaction commits and stops when Kafka acknowledges the publish, both read on the same machine — which is why the database and the broker share this host.
Throughput
Up to the ceiling the relay delivers one event for every event offered and the backlog stays flat — a handful of rows pending at any moment, whatever the rate. Past it nothing fails and nothing is dropped: the excess accumulates in the outbox and drains once the offered rate falls back.
What sets the limit
What runs out at the ceiling is CPU: it reaches 94% of both cores while disk utilisation sits under 1% and the relay waits on I/O for a tenth of a percent of its time. The bottleneck here is computation — encoding, publishing and polling — not the rate at which the database writes to disk.
Scenarios
A pass means zero ordering violations and zero lost events — never a throughput threshold. One row is marked HOST: the scenario hit a limit of this two-core VM, not of Tandem.
| Scenario | What it does | Result | Observed |
|---|---|---|---|
| Sustained throughput | Finds the highest rate the relay holds | PASS | 1250 events/s, ceiling bracketed below 1300/s |
| Latency at normal load | Holds 600 events/s and times every event | PASS | p50 12.8 ms, p99 90.9 ms |
| Hot partition | Sends 80% of traffic to one aggregate | PASS | 985 rows queued behind the hot bucket; the rest kept flowing |
| Saturation | Drives deliberately past the ceiling, then stops | HOST | Backlog drained, but two cores need longer than the 9-minute window; nothing lost or reordered meanwhile |
| Worker failover | Kills the relay mid-flight and restarts it | PASS | 0 duplicates against a bound of 800 |
| Poison message | Injects an event that can never be delivered | PASS | Its aggregate blocked; every other aggregate unaffected |
| Multi-instance coordination | Runs three relays and kills one outright | PASS | Survivors re-took its 84 buckets, 256 of 256 covered, no duplicates |
Endurance
Every scenario above measures the first few minutes of a relay's life. The ways an outbox actually fails in production are slow ones — a leak, index bloat on a table churning through millions of rows, autovacuum falling behind, a lease renewal path that misbehaves only after many cycles. None of them shows up in four minutes.
So Tandem is also run long. This one run predates the adaptive idle backoff, so its latency figures below carry the fixed 100 ms poll interval the numbers further up no longer use; what it is evidence for — drift, or its absence, over six hours — is unaffected.
A fixed 400 events/s held for six hours under two-instance lease coordination, with retention set so the relay deletes delivered rows throughout — the churn a real deployment lives in, not a table that only grows. 8.6 million events, reported in eighteen twenty-minute windows so the last can be compared against the first.
Throughput 395.7 → 399.6 events/s against 400 offered; p50 flat at 58–60 ms and p99 within 156–173 ms, with no slope in either.
Ownership complete and disjoint in every one of the eighteen windows, across roughly 720 lease renewal cycles.
Zero duplicates too, over the whole run. The outbox reached a steady size instead of growing, and autovacuum kept pace with the churn.
This one ran on a developer machine rather than the VM described below, so read it for the absence of a slope rather than for a rate — the whole value of an endurance run is in what does not change. Windows, host samples and the run's own summary are in docs/benchmark-results/2026-08-30-endurance.
Recovery
What an outage leaves behind is a backlog: the rows sit in the outbox and wait, and the only
question is how long they take to drain. The claim reads pending rows in id order through a
partial index of their own (schema v5), so catching up costs what the backlog
costs rather than what the table holds.
At 400 events/s that is ~713 000 undelivered rows. A 15-minute outage, ~358 000 rows, clears in 146 s.
On the 2 vCPU host, 239 871 rows cleared at 3 656 rows/s and 479 328 at 3 440 — the same rate at twice the backlog.
What the index costs the write path, measured in WAL bytes. Zero ordering violations, zero lost and zero duplicates at every backlog.
The recovery times come from a developer machine; the reference-host pair confirms the same behaviour on native Docker. Runs, plans and samples are in docs/benchmark-results/2026-08-31-outage-recovery.
Test environment
| Host | AWS m7i-flex.large — 2 vCPU, 8 GB, Ubuntu 24.04, native Docker,
eu-central-1 |
| Topology | PostgreSQL, Kafka, the relay and the load driver on one machine, so COMMIT → ack is measured against a single clock |
| Relay | 8 workers, batches of 100, 256 buckets |
| Events | 1 KB payloads |
| Producer | acks=all, enable.idempotence=true — the production-mandated
settings, never a faster configuration for the sake of a number |
Treat these as a floor. Latency is stable on this host; sustained throughput is not: this is a burstable instance, whose capacity moves with recent CPU use, and its ceiling has ranged from 725 to 1450 events/s across sessions. Those two cores also carry PostgreSQL, Kafka and the load driver alongside the relay, so a host with cores of its own should do better on both counts.
Every figure is backed by its raw run — logs, resource samples and the script that redraws these charts from them — in docs/benchmark-results. The methodology is in HLD-load-testing.md, the search algorithm in LLD-benchmark.md §7.
Tandem