The Cron Job That Ran Thirteen Times in One Night

The job was a nightly reconciliation, scheduled for 02:30, idempotent in the sense that running it twice was harmless but running it thirteen times produced thirteen reports and one very confused on call. The alert that fired was not about failure. It was about a metric that counts runs, spiking to thirteen.

The night in question was a daylight saving transition. That fact was in the incident from the start and was ignored for ninety minutes, because nobody wants the answer to be "the calendar".

This was a containerised cron on a host whose local timezone observed DST, running a job that assumed it would be scheduled in a timezone that does not jump.

The symptom, precisely

The job is meant to run once per day. On the transition night it ran thirteen times between 02:00 and 04:00 local, then did not run at all the following night. Two different failure modes from one root cause, which is the signature of a clock problem: the same defect produces too many runs in one direction of the jump and zero in the other.

The hypotheses that died

The scheduler is misfiring

First suspect was the cron daemon itself, a fork loop or a stuck timer. I pulled the daemon's own logs and the system's monotonic timestamps. The daemon believed it was doing the right thing given the wall clock it was reading.

It died because the scheduler was faithfully executing against a clock that had lied to it.

Something is retrying the job

Next I assumed a wrapper with retry logic was resubmitting. The runs were not retries of a failed attempt. Each run started fresh, at a distinct wall clock time, and each succeeded. There was no retry loop in the code.

It died because the runs were scheduled, not retried. The trigger was the clock reaching 02:30 repeatedly.

A duplicate deployment

Then I checked for two instances both running cron. There was one instance. The thirteen runs all came from the same host and the same PID lineage.

It died because there was no second scheduler. One scheduler, one clock, many firings.

The breakthrough

On the spring forward transition, local time jumps from 02:00 to 03:00, skipping the 02:30 the job is meant to hit. On the fall back transition, local time repeats the 02:00 to 03:00 hour, so 02:30 happens twice. Neither of those is thirteen.

The thirteen came from the container's view of time fighting the host's view. The container had a stale or differently configured timezone database and was converting the host's UTC ticks into a local time that wandered during the transition, re evaluating "is it 02:30 yet" against a clock that oscillated. Each oscillation that crossed the boundary fired the job. The exact count is an artefact of the tzdata mismatch and the polling interval, but the shape, many firings clustered around the transition, is the fingerprint.

The following night's zero runs is the same defect: the container's conversion placed 02:30 at a moment the scheduler had already passed, so the window was skipped.

This is the scheduling version of the timezone bug that corrupted analytics: the data was fine, the interpretation of instants was not.

What I changed

Run the scheduler in UTC. The single highest value change. Cron expressions evaluated in UTC never see a DST jump, because UTC does not have them. The job can still do local aware work inside, but the trigger lives in a timezone with no transitions.

Make the trigger an instant, not a wall clock time. Where the system allows it, schedule by interval or by a UTC timestamp and compute the local presentation separately. A wall clock time is a label, and labels are what jump.

Deduplicate by business day, not by process. The job now writes a marker for the logical day it processed, and refuses to reprocess a day that already has a marker. This turns a thirteen fold scheduling failure into one real run and twelve cheap no ops, so the next clock defect is silent instead of loud.

Align tzdata across host and container. The oscillation came from mismatched timezone databases. Pinning the same tzdata in the base image and verifying it at build time removed the disagreement that made the transition chaotic rather than merely wrong.

What I would do differently

I would have asked "what night is it" before "what is broken". Time based anomalies that coincide with a transition are transition problems until proven otherwise, and the proof is cheap: reproduce the schedule against the transition in a test with a fake clock.

I would also have made the job's idempotence about the business day from the start. "Running it twice is harmless" is a property you should not have to rely on, because some clock, somewhere, will eventually run it thirteen times.

If it happens again

Schedule triggers in UTC, make repeated execution a no op by logical period, and treat any run count anomaly near a DST transition as a clock interpretation bug, not a scheduler bug. The scheduler is usually doing exactly what the clock told it, and the clock is the only component in the stack that is allowed to travel in time.

One more habit earns its place: log the scheduler's own view of the current instant on every evaluation, in UTC and in local, side by side. When the next transition anomaly arrives, that log line shows the oscillation directly, and the ninety minute argument becomes a one minute read. The defect in this incident was never hard to understand. It was hard to see, and the difference between those two is what observability is for.

The same "two components disagree about what time it is" defect shows up in tokens, covered in JWT expired and clock skew, where the jump is seconds rather than an hour but the lesson is identical.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud