A daily job in our unattended pipeline is configured to run at six in the morning. For a long time I described it that way in my own notes, in the runbook, and to anyone who asked: it runs at six. That sentence was wrong in a small way that turned out to matter. The configuration is a request submitted to a scheduler we do not own. What the scheduler does with that request depends on how much other work it is holding at that hour. In our own operation, the actual start time drifted by tens of minutes depending on how busy the platform's scheduler happened to be.
Tens of minutes sounds harmless, and in isolation it is. Nothing in our pipeline breaks because a job starts at six thirty-one instead of six. The damage comes from the sentence, not the delay. Once you write a fixed time into a config file, you start reading that time back out as if it were an observation. Every downstream assumption inherits the confidence of a number you typed rather than a number you measured.
Here is the shape that assumption takes. A second step is scheduled shortly after the first, on the theory that the first will be finished by then. A report is generated on the belief that the day's inputs have already landed. A window of time is described as "yesterday" by subtracting twenty-four hours from a start time nobody verified. None of these are unreasonable designs. They are all quietly built on the phrase "at six," and the platform only ever promised "close to six." Those two phrases live in the same slot in the config and have completely different guarantees behind them.
What made this hard to notice is that drift produces no failure to look at. A job that starts late still starts. It still writes its pass line to the summary. If the downstream step happens to find what it needs, the whole chain reports success, and the only trace of the drift is a timestamp nobody opens. The bad day arrives when the drift is slightly larger than usual and the ordering finally inverts, at which point the failure shows up somewhere other than where the cause is. The report is empty, or the second step processes the previous day's material, and the investigation starts from the wrong end of the chain.
The correction we made is boring and cheap. We record the time the job actually began, separately from the time it was supposed to begin, and we keep the difference as its own field. That single number is now a signal instead of a footnote. It gives us a baseline for what normal drift looks like on this platform at this hour, which means an unusual value becomes visible before it causes an inversion. It also converts a belief into a measurement, which is the only real change here. We stopped asking the config what time the job ran and started asking the job.
The second correction is about how the steps relate to each other. Whenever a later step depends on an earlier one finishing, the dependency belongs in the work itself, not in the clock. If step two needs step one's output, it should be looking for that output, not for the hour hand. Scheduling two things near each other and hoping is a coordination strategy that works until the scheduler is busy.
The general lesson I keep relearning in unattended operation is that every number in a config is a request, and every number in a log is a measurement, and the two look identical when printed side by side. Human operators fill in the difference automatically because they are standing there watching the thing start. With nobody standing there, the only knowledge you have is the knowledge you recorded on purpose. Six in the morning was never a fact about our system. It was a preference we stated once and then read back to ourselves for months.
