While the Alert Is Still Firing, Keep an Incident Timeline Like This

At 10:04 UTC the alert fires. By 10:40, five people are in the chat. Then someone new joins and asks two simple questions: "When did it start? What have we tried?" Nobody can answer without scrolling back through 200 messages.

A timeline fixes that. It's one plain list of times and facts, written while the incident is still going. An incident is any unplanned problem that hurts a service enough that people stop other work to fix it.

Open one plain file first

Before you run a single command, start the file. A text file, a shared doc or a chat thread pinned to the top all work. The Google SRE book calls this a "living incident document". It says the document can be messy, but it has to be useful, and it should be easy for several people to edit at once.

Give it a name with the date, like incident-2026-11-30.md. Put three lines at the top and keep them current:

1. Status: investigating, fixing, watching, or resolved.
2. What users see: one sentence.
3. Who is doing what: a name or role next to each task.

Add a line every time something happens

Each line gets a time and one fact. If you work in a terminal, a tiny shell function saves typing:

note() {
  printf '%s  %s\n' "$(date -u +%H:%MZ)" "$*" >> incident-2026-11-30.md
}

Now note restarted worker-2 adds a line like 10:22Z restarted worker-2 to the end of the file. Piece by piece:

• date -u uses UTC, and +%H:%M prints the hour and minute, per the GNU coreutils manual. The Z is a common way to mark a UTC time, and GNU date reads Z as UTC too.
• "$*" is everything you typed after note.
• >> adds to the end of the file and never wipes what's already there.
• If a note has an apostrophe or a symbol like > or ;, wrap it in double quotes, like note "rolled back yesterday's deploy". Otherwise the shell reads those characters itself.

Write down four kinds of things:

1. What you saw. "Error rate on the order page is at 8%."
2. Each action you took. "Rolled back to yesterday's version."
3. What you decided, and why. "Not restarting the database yet: other apps share it."
4. What changed afterwards. "Errors dropped to 1% two minutes after the rollback."

Put every time in UTC

People join from different cities. If one writes "3:15" and another writes "10:15", the timeline is useless an hour later. Pick UTC for everything, including times copied from dashboards and logs. If a tool shows local time, convert it before you paste.

Mark guesses as guesses

During an incident you'll think things you can't prove yet. Write them, but label them:

10:31Z  GUESS: the new image cache is filling memory on worker-2
10:36Z  CONFIRMED: worker-2 memory at 97%, other workers at 40%

Later, nobody has to work out which lines were facts and which were hunches.

Here's a short example

This is a made-up incident on a toy parcel-tracking app:

Status: resolved   Users saw: tracking pages slow or blank
10:04Z  ALERT: tracking page p95 latency above 3s
10:07Z  on-call lead takes point; second engineer reads logs
10:12Z  SAW: worker-2 log full of "out of memory" since 09:58Z
10:18Z  DID: took worker-2 out of the load balancer
10:21Z  SAW: latency back under 1s
10:26Z  DECIDED: roll back 09:50Z deploy (image cache change)
10:33Z  DID: rollback finished; worker-2 back in
10:48Z  WATCHING: 15 min clean, closing the incident

"p95 latency" means 95% of requests took that long or less. A load balancer spreads visitors across several servers, so taking one out stops it from getting traffic.

Turn it into a short review the next day

Once things are calm, the timeline becomes the backbone of a review (Google calls it a postmortem): a short write-up of what happened and what to change.

1. Copy the timeline in as it is. Fix typos, nothing more.
2. Write the impact. Who was affected, for how long. Here: about 23 minutes of slow or blank tracking pages, from the first out-of-memory line at 09:58Z to 10:21Z, when latency recovered.
3. List what helped. The log made the bad worker obvious within 8 minutes.
4. List what got in the way. No one knew who had made the deploy at 09:50.
5. Write two or three follow-ups. Each one gets an owner and a date.

Keep the review blameless

The SRE book says a blameless postmortem looks for the causes of an incident without blaming any person or team. It assumes people acted in good faith with what they knew at the time.

In practice, write "the deploy had no memory test" instead of "the person who deployed didn't test it". The first one leads to a fix. The second one teaches people to hide mistakes next time.

Where this fits at DevGrid Staffing LLC

DevGrid Staffing LLC works with SRE and DevOps teams, where on-call shifts are a normal part of software engineering. A tidy timeline is something anyone on the rota can keep, and it makes the review the next day much easier.

One thing to try today

Paste the note function into your shell, run note testing my timeline, and check the file. Next time an alert fires, start the file before you start fixing.

Written at the DevGrid Staffing LLC workbench for people on SRE and pipeline teams · devgridstaffing.com