How a long job's write-back silently stomped a pause

Here is the bug in one sentence, so you can carry it before you read the story: if an operation reads a piece of mutable state when it starts, runs for a long time, and then writes that same state back when it finishes, it will silently erase any change that arrived during the run. The longer the operation, the wider the window, and the more certain the loss.

I hit this in my own scheduler. The shape is a textbook lost update, but the gap between the read and the write was not a few microseconds of CPU. It was minutes, because the work in the middle was a full agent task. That is what made it real rather than theoretical.

The setup

The scheduler runs jobs. Each fire goes through one function: pick the job up, call into the runtime to do the work, then write the result back to the row. At pickup, the function takes a snapshot of the job, including its status, which at that moment is active. It carries that snapshot through the run and uses it to decide what to write at the end.

For an ordinary cron tick the run is fast and the snapshot is fresh enough. But an agent task is not fast. The runtime call can take minutes. Meanwhile the scheduler is still live, still serving the rest of the system, and one of the things it serves is a pause.

An operator pauses a job. Pause is a single write: set status = 'paused' on the row, clear the next run. That write lands cleanly. The row is paused. Then the long-running job that was already in flight finishes, reaches its write-back, and stamps the row with the snapshot it took at pickup. The snapshot says active. The pause is gone. The job will fire again as if nobody touched it.

Why it stays hidden

Nothing throws. The pause succeeds, the run succeeds, the write-back succeeds. Every individual operation is correct in isolation. The defect lives only in the ordering, and only when the pause happens to fall inside the run window. Pause a fast job and you will never see it, because the window is too small to aim at. Pause a slow one mid-run and the loss is reliable.

That is the trap of a stale snapshot. The value you read was true when you read it. It is not true anymore, and the code has no way to notice, because it never looks again. It trusts a fact it gathered minutes ago and writes it down as if it were still current.

The fix

The instinct is to reach for a lock, and you can, but the cheaper and more honest fix is to stop trusting the stale read at the one place it matters. Right before the write-back, re-read the live status from the row. If it changed under you, honor what is there now instead of what you remembered.

// handleMessage for an agent task runs for minutes, and pauseJob() can
// land during that window. The newStatus we computed came from the
// snapshot taken at pickup, so an unconditional write-back would stomp a
// concurrent pause. Re-read the live status and honor an external pause.
if (newStatus === "active") {
  const live = db.query("SELECT status FROM scheduled_jobs WHERE id = ?")
    .get(job.id);
  if (live?.status === "paused") {
    newStatus = "paused";
    nextRunAt = null;
  }
}

The scope matters, and it is the part that is easy to get wrong. I only defer to the live row when my own outcome was active, meaning "carry on, schedule the next fire." If this run finished or failed, those are terminal verdicts this execution earned, and they must win over a pause that raced in. A completed job is completed. So the re-read guards exactly one case: the continuation. Everything else writes through.

Clearing the next run on the deferred pause is safe because resuming recomputes it. When the operator un-pauses, the resume path builds a fresh schedule from scratch. There is nothing to preserve by leaving a stale timestamp behind, and leaving one would mean a paused job still carried a pending fire.

Proving it goes red

A test that only ever passes proves nothing. So I made the regression test mutate the row from inside the mocked runtime call, exactly the way a real pause would land mid-run: the mock flips the status to paused while handleMessage is pretending to work, then the run completes and hits the write-back. With the fix in place the test asserts the row stays paused with no next fire. Then I stashed the fix, reran, and watched it go red, the write-back stomping the pause back to active. Restored the fix, green again. The stash bisect is the evidence that the guard actually guards something.

The rule I took away

Read-modify-write is only safe when the modify is short. Stretch the middle and the read goes stale, and an unconditional write-back turns into a lost update aimed at whoever wrote during your window. Two fixes work. Re-read the state at the write boundary and reconcile, which is what I did here. Or narrow the write so you only touch the columns you actually own, and never write back a field you merely read. The first is right when one field can be authored by two parties. The second is right when you can carve the row so no field has two authors. Either way, the question to ask of any long operation is the same: between when I read this and when I write it, who else could have changed it, and what happens to their change when I save mine?