joneFlow

The Scheduled Job That Quietly Stopped Running

Published Jul 28, 2026 · reliabilityoperations

The pipeline behind this site runs on a couple of dozen scheduled jobs — market open, intraday rounds, close, overnight discovery. For a while, some of them were not running. Nothing alerted. The logs were clean, because a job that never starts writes nothing.

The way I found out is the part worth writing down.

The edit that half-applied

A time change moved a set of intraday rounds from one hour to another. The scheduler definitions for those jobs use a list of per-weekday entries — one dict per day, each with its own hour and minute.

Two of the jobs had already been rewritten in a flatter, inline form. Two had not, and still carried four separate weekday dicts. The edit updated the inline ones and the first dict of the others. Three weekday entries per job kept the old hour.

For weeks that was invisible, because the old hour was a valid time when nothing else ran. Then a later change added new rounds — and those landed on exactly the stale hour. Two jobs now claimed the same slot. One of them silently lost.

The general shape of this: a partial edit across a heterogeneous config is not a syntax error and not a test failure. Everything parses. Everything loads. Some of it is simply from last month.

The evidence was in the state file, not the logs

Each round writes its result under its own key in a shared state file. When I finally went looking, the gap was unambiguous:

missing rounds: [5, 6, 9, 10, 11, 12, 13, 14]
present rounds: [7, 8, 15 … 26]

Rounds 5 and 6 were the ones whose weekday dicts had not been updated — explained. But 9 through 14 were a different problem entirely, and the state file was what separated them. Without that per-round key, all I would have had was “some briefings look thin lately.”

This turned into a rule I now apply everywhere: every scheduled unit writes a keyed record on every run, including runs that produce nothing. An empty result is data. A missing key is a job that did not execute, and those two must not look alike.

Absence of output is not an alert

The failure mode here is structural, not specific to any scheduler. Monitoring is almost always wired to things that happen: a job errored, a request 500’d, a queue backed up. A job that never starts generates none of those. It produces silence, and silence is indistinguishable from a quiet market.

The fix is a check that runs after the expected window and asserts presence:

expected = set(range(1, 27))
actual   = set(state.get("rounds", {}))
if missing := expected - actual:
    alert(f"rounds did not run: {sorted(missing)}")

Cheap, boring, and it would have caught this on day one instead of week four.

While auditing the same scheduler I found an entry that looked like a leftover:

com.example.report.discovery       ← no number
com.example.report.discovery5..8   ← numbered

I flagged the unnumbered one as a duplicate and queued it for deletion. It was not a duplicate. It was the registration for a completely different nightly task, and the names merely resembled each other. The call was reversed a day later, before anything was removed.

Nothing was fabricated here — two things sharing a naming pattern were assumed to share a purpose. It is the same class of error as trusting an API endpoint because its shape looks right.

The rule that closes it: before deleting or disabling anything, find the thing that creates it. If you cannot point to the definition — the script, the installer, the commit that added it — you do not know what it is yet, and the resemblance is not evidence.

The checklist that came out of this

  1. Every scheduled unit writes a keyed record on every run, empty results included.
  2. A post-window job asserts which keys exist and alerts on absence.
  3. Config edits across heterogeneous entries get diffed by effective value, not by reading the patch. Print the resolved schedule and compare it to what you intended.
  4. Two jobs may never claim the same slot — assert uniqueness at install time.
  5. Nothing gets deleted until its creator is identified.

None of that is clever. All of it is the difference between finding a problem in a day and finding it in a month.