The rollout started at two.
Ten percent of traffic
on the new version.
Your job was simple.
Watch the error graph.
If it moves,
roll back.
You watched it.
You really did.
At ten to four
someone from support asked
why checkouts had been failing
since twenty past three.
You scrolled back.
There it was.
A clear step up
on the graph you had been looking at.
You do not remember seeing it.
You remember the graph.
You remember the coffee
going cold beside it.
You do not remember
the moment the line moved.
That is not laziness.
It is one of the oldest findings
in the study of attention.
During the Second World War,
radar operators watching for submarines
were missing contacts,
and the misses came
later in their watches.
In 1948
Norman Mackworth
built a test to study it.
A pointer moved around a clock face
in small steps.
Every so often
it jumped twice as far.
People had to report the double jumps,
for two hours.
Within the first half hour,
they were already
catching noticeably fewer.
He called it
a decline in vigilance.
Watching for something rare
is not resting.
It is hard work,
and the brain
quietly stops doing it.
A rollout graph
is a clock face
where nothing happens
most of the time.
So stop asking people
to stare.
Write down,
before the rollout starts,
the exact thing you are watching for.
Error rate above this number.
Latency above that one.
Then turn that into an alert,
so a machine does the staring.
If a person has to watch,
give them a shift
of half an hour
and then hand over,
with the numbers said out loud.
And check on a timer,
on purpose,
rather than glancing
whenever you remember.
Attention is not a camera
you point and leave running.
It fades,
and it does not tell you
when it has gone.
Plan for that.
– Serguey Asael Shinder
