The Cron Job That Overlapped Itself and Doubled the Work

I started at the incident, because that is where the symptom landed, and walked it backwards link by link until the corrupt stopped looking innocent.

Here is the shape of the self-overlapping scheduled job, and the one property that makes it intermittent: the job usually finishes inside its interval, so the overlap only happens on the slow run, and the slow run is the one nobody planned for.

The case was a nightly export on a cron schedule; the mechanics generalise to any scheduler that starts the next run without checking whether the previous one finished.

Why the overlap happened

The schedule was the time based, and the time based was the fire at the hour, and the hour was the unconditional, and the unconditional was the no check, and the no check was the running instance's ignorance, and the ignorance was the second start, and the second start was the concurrent, and the concurrent was the two writers, and the writers were the file.

The duration was the variable, and the variable was the data's growth, and the growth was the slow, and the slow was the beyond the interval, and the beyond was the overlap, and the overlap was the eventual, and the eventual was the growth's certainty, and the certainty is the reason the overlap is the when rather than the if.

The cron is the not the supervisor, and the not supervisor is the no awareness, and the no awareness is the fire and forget, and the forget is the overlap, and the overlap is the incident, and the incident is the reason the lock is the job's responsibility rather than the scheduler's.

Why the file was corrupt rather than duplicated

The two instances wrote the same path, and the path was the file, and the file was the shared, and the shared was the interleaved, and the interleaved was the half of each, and the half was the corrupt, and the corrupt was the worse than the duplicated, because the duplicated is the detectable and the corrupt is the plausible, and the plausible is the downstream's error, and the error is the incident's spread.

The write was the not atomic, and the not atomic was the open and the write and the close, and the three were the interleaved with the other instance's three, and the interleaved was the mixed content, and the mixed was the unreadable, and the unreadable was the downstream's failure, and the failure was the incident's reach.

This is the same shared state hazard as reviewing shared state by asking what happens under contention, same mechanism, different costume.

The fix, in order

Added a lock file with a stale timeout. The instance takes the lock, and the lock is the second's skip, and the skip is the fix, and the fix is the flock or the pidfile, and the pidfile is the stale's handling, and the handling is the timeout, and the timeout is the crashed instance's release, and the release is the discipline, because the lock without the timeout is the permanent skip.

Wrote to a temporary file and renamed. The write is the temp path, and the temp is the complete, and the complete is the rename, and the rename is the atomic, and the atomic is the no partial, and the no partial is the fix, and the fix is the discipline, because the rename is the single operation and the readers see the old or the new and never the half.

Alerted on the duration. The job reports its runtime, and the runtime is the trend, and the trend is the growth, and the growth is the interval's approach, and the approach is the warning, and the warning is the fix, and the fix is the metric, and the metric is the discipline, because the overlap is predictable from the duration trend.

Made the job idempotent per period. The run records the period it processed, and the processed is the second's skip, and the skip is the fix, and the fix is the key, and the key is the date, and the date is the deduplication, and the deduplication is the discipline, because the lock prevents the concurrent and the key prevents the repeat.

Separated the schedule from the work. The cron triggers the queue, and the queue is the single consumer, and the consumer is the no overlap, and the no overlap is the fix, and the fix is the architecture, and the architecture is the discipline, because the queue serialises what the cron cannot.

If it happens again

A time-based scheduler starts the next run without checking whether the previous one finished, so a job that outgrows its interval overlaps itself. Take a lock with a stale timeout, write to a temporary file and rename, alert on the duration trend, make the run idempotent per period, and put the queue between the schedule and the work.

The export file came out half one run and half the next, because the job had grown slower than its interval and the scheduler started the second instance while the first was still writing to the same path. I now check for that property first, because it is where the failure actually lives.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud