SIGSEGV at 03:14 — when forbid(unsafe_code) doesn't save you

The first page came in on a Tuesday. 03:14 AM local. Not 03:00, not 03:30. 03:14, for four nights in a row.

The journal entry that woke me up:

[2026-04-22T03:14:07.882Z] worker[7]: received signal: SIGSEGV (11)
[2026-04-22T03:14:07.882Z] worker[7]: fault address: 0x7f9c1afc0000 (not mapped)
[2026-04-22T03:14:07.882Z] worker[7]: dumping core to /var/crash/inventory-7.core.gz
[2026-04-22T03:14:08.014Z] worker[7]: exit code 139
[2026-04-22T03:14:08.014Z] systemd[1]: inventory-svc.service: Main process exited
[2026-04-22T03:14:08.015Z] systemd[1]: inventory-svc.service: Scheduling restart in 5s.

The service is in Rust. Seventeen crates in the workspace, every one with #![forbid(unsafe_code)] at the top. Rust services crash, but they crash so seldom that this one woke the whole team within an hour. By the third night we had a war room.

What we knew on Wednesday

Four facts, harvested through three nights of paging:

1. It only happened in production. Staging ran the same binary, same config, same Postgres dump, same Kafka topics. Nothing.
2. It always happened at 03:14. Not at 14:00 under our synthetic load. Not at 02:50. Never within 90 seconds either side of 03:14.
3. The crashing worker rotated. Whichever of the eight workers happened to pick up the request at 03:14 died.
4. The work was always the same shape: a BatchReconcile from the warehouse adapter.

Those four facts contained the entire answer. We didn’t see it for another 48 hours.

The obvious passes

We did the dumb things first. Cranked the log level to trace on the relevant code path, deployed, waited until 03:14. The worker crashed and the trace logs ended one line before the SIGSEGV. We learned that the crash happens inside a span where structured logging is buffered, which is to say we learned nothing useful.

We tried to reproduce by replaying the production Kafka log from 03:13:00 through 03:15:00 against a fresh staging worker. The staging worker handled it without complaint. We tried again with the partition order shuffled. Same. Tried a third time after wiping the worker’s local state. Same.

We attached gdb to a running production worker in non-stop mode at 03:13:50, hoping to break in the moment the crash happened. The crash did not come. Detaching gdb and letting the worker run untouched, it crashed at 03:14:08 the next night. The debugger was slowing the worker down enough that the offending request finished outside the crash window.

The corefile

Thursday morning, finally, a usable corefile. Our infra team had set up automatic upload to an S3 bucket on every crash. If you do not have that, set it up tonight; it pays for itself the first time something does not reproduce locally.

$ gdb ./inventory-svc /tmp/inventory-7.core
[New LWP 18341]
[New LWP 18342]
...
Reading symbols from ./inventory-svc...
Core was generated by `./inventory-svc --config=/etc/inventory.toml'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f9c1a8b2e10 in core::ptr::read_unaligned<inventory::reconcile::Request> ()
    at /rustc/.../library/core/src/ptr/mod.rs:1284
1284    pub const unsafe fn read_unaligned<T>(src: *const T) -> T {
(gdb) bt
#0  core::ptr::read_unaligned<inventory::reconcile::Request> at ptr/mod.rs:1284
#1  warehouse_rpc_codec::codec::decode_batch at src/codec.rs:142
#2  inventory::worker::handle_batch_reconcile at src/worker.rs:412
#3  inventory::worker::dispatch_loop at src/worker.rs:312
#4  tokio::runtime::task::raw::poll<...>
...

read_unaligned. Inside warehouse_rpc_codec. Inside an external crate.

The dispatch loop we’d been staring at for two days was a red herring:

struct DispatchQueue {
    inbox: VecDeque<RawFrame>,
    deadline: Option<Instant>,
}

impl DispatchQueue {
    async fn run(&mut self) {
        loop {
            tokio::select! {
                frame = recv_from_kafka() => self.inbox.push_back(frame),
                _ = sleep_until(self.deadline) => {
                    if let Some(frame) = self.inbox.pop_front() {
                        spawn(handle_request(frame));
                    }
                }
            }
        }
    }
}

&mut self everywhere. The borrow checker will not let two tasks touch the deque concurrently. We had spent a day and a half asking ourselves how the queue was being corrupted, when in fact the queue was fine. The frame came out of the deque intact and got handed to handle_request, which called into the codec, which crashed.

Following the breadcrumbs

That evening, in Cargo.lock:

[[package]]
name = "warehouse-rpc-codec"
version = "0.4.2"
source = "git+https://example.invalid/warehouse-rpc-codec.git#9a14bb7c"

A git dependency. Not pinned to a release tag. Commit 9a14bb7c was 11 weeks behind the codec team’s master branch.

$ git -C ~/src/warehouse-rpc-codec log --oneline 9a14bb7c..HEAD
b822c91  Fix: BatchReconcile decoder mis-aligns on 32+ requests
4a3f1ad  Use Vec::with_capacity to avoid re-allocs
2dd1e92  Decode in place via unsafe pointer math   <-- !!
1a2b3c4  Bump tokio
...

2dd1e92 had introduced raw pointer arithmetic into the decoder. Three weeks later b822c91 had fixed the alignment bug it introduced. We were pinned at a point in history that contained the unsafe but not the fix.

The relevant function in 0.4.2:

// warehouse-rpc-codec v0.4.2
pub unsafe fn decode_batch(buf: &[u8]) -> Vec<Request> {
    let count = u32::from_le_bytes(buf[0..4].try_into().unwrap()) as usize;

    // Slow path: simple, correct, untouched by 2dd1e92.
    if count <= 31 {
        return decode_slow(buf);
    }

    // sizeof(Request) is 40 bytes after #[repr(C)] padding; the wire format
    // packs items at 36 bytes each. Walking the pointer by sizeof(Request)
    // drifts 4 bytes per item. By item 32 the read straddles the end of buf
    // and starts pulling from whatever follows it in memory.
    let mut out = Vec::with_capacity(count);
    let ptr = buf.as_ptr().add(4) as *const Request;
    for i in 0..count {
        out.push(std::ptr::read_unaligned(ptr.add(i)));
    }
    out
}

For count = 32 the final read starts 128 bytes past the end of buf. Most of the time that’s inside the next allocation on the same arena. You get a corrupted Request, the upstream caller rejects it with InvalidArgument, and the only visible effect is a sporadic warning in the logs. (We had been seeing exactly those warnings for weeks, attributed to a noisy upstream.) Occasionally buf is the last live allocation on its page, the drift carries the pointer into an unmapped page, and you get a SIGSEGV.

#![forbid(unsafe_code)] applies to the crate it sits in. Dependencies do whatever they like.

Why 03:14

That still left the timing. Why exactly 03:14?

The fast path only runs at count > 31. So: where does that workload come from?

BatchReconcile is the rollup that fires after the European-region pickers shut their terminals for the day. They go offline at 03:00 UTC, and the rollup waits a 14-minute drain window to let in-flight scans settle. The first BatchReconcile after the drain has accumulated whatever was pending; typically 32 to 60 items.

Every other BatchReconcile during the day stays at count ≤ 31 because real-time picker traffic keeps the queue short. Only the post-drain rollup ever crossed 32. Once a day. At 03:14.

Staging didn’t have the rollup configured. Local didn’t either. The only place the input shape that triggered the fast path ever existed was prod, at exactly that minute.

The fix

Ten minutes:

[dependencies]
warehouse-rpc-codec = { git = "https://example.invalid/warehouse-rpc-codec.git", tag = "v0.5.1" }

A version with the alignment fixed. We rebuilt, deployed, waited. 03:14 came and went without a page, three nights in a row.

The meta-fix

That took longer.

cargo deny now refuses unpinned git dependencies in CI. The allowed forms are tag = "..." or branch = ... + rev = ...; anything else fails the build. The first run after we landed this rule failed on four crates other teams had imported. We fixed them.

Nightly staging now replays the trailing 24 hours of production Kafka with realistic traffic shaping (replay-rate, partition fanout, gaps). It would have caught this bug on night two if we’d had it then.

cargo geiger runs on every dependency bump and reports new unsafe introduced in transitive deps. It’s noisier than I’d like. The signal-to-noise is still vastly better than another four-night incident.

And there’s a soft rule: anyone bumping warehouse-rpc-codec (or any other crate whose geiger report is non-empty) skims the diff since the last pin. We’ve done it three times since April. We caught one regression worth catching and approved two changes without ceremony.

Things worth carrying

• forbid(unsafe_code) is a property of your crate, not your binary. Your dependency graph is part of your trusted computing base, whether you treat it that way or not.
• Pin everything. Floating versions, especially git deps without tag or rev, are an outage with a delayed fuse.
• Crashdumps are not optional infrastructure. If you can’t get a corefile out of prod, you cannot debug what only happens in prod.
• A timing-locked bug almost always has a deterministic driver behind it. Find what fires on that clock and you have found the bug. Cron jobs, market open/close, batch windows, log rotation, GC pauses; clock-shaped things.
• If gdb -p makes the symptom go away, the bug is timing-related. Reach for rr or valgrind next.

The five-line cargo deny rule that would have prevented this is now in deny.toml, with a comment pointing at this post.

Greg Law

If you want a real tour of post-mortem debugging on Linux, find Greg Law’s CppCon talks. Give me 15 minutes and I’ll change your view of GDB (CppCon 2015) is the short version. GDB - A Lot More Than You Knew (CppCon 2016) is the one you watch on a Saturday with a notepad open. Both are on the CppCon YouTube channel.