The worst moment in incident response is not discovering you were breached. It is discovering you have no idea what the attacker did, because the logs that would have told you were never collected, or rolled off three days ago, or existed only on the machine that got wiped. I have sat in that room. The question "what did they access" hangs in the air, and the honest answer is "we cannot know," which is somehow worse than any specific bad answer would have been.
Logging is one of those disciplines that feels like pure overhead right up until the day it is the only thing standing between you and total blindness. Nobody gets promoted for retaining authentication logs. It costs storage, it costs setup, and ninety-nine days out of a hundred nobody reads them. Then the hundredth day arrives, and the entire quality of your response, what you can contain, what you must disclose, whether you can even tell customers the truth, depends on decisions you made about logging months before you knew you would need it.
The mistake I see most is logging in a way that helps you operate but not investigate. Application logs full of debug noise, but no record of who authenticated from where, which admin actions ran, or what data was queried. When an incident hits, that operational logging is nearly useless, because it was designed to help you fix bugs, not to reconstruct what a determined intruder touched across weeks. Those are different questions and they need different records, kept somewhere the attacker cannot quietly edit.
That last part matters more than people expect. Logs on the compromised host are logs the attacker can delete. The first thing a competent intruder does is cover their tracks, which means your security logs need to ship somewhere separate, append-only, out of reach of the credentials they stole. A tamper-proof trail on a system they never breached is worth more than gigabytes of everything on the box they own.
You cannot investigate what you did not record. Decide now, on a calm day, what you would desperately want to know after a breach, and make sure you are collecting it, retaining it long enough, and keeping it somewhere the attacker cannot reach. Your future self, in that terrible room, is depending on it.
– Serguey Shinder
