A database server that had not changed in a year was running about twenty percent busier than it did the previous autumn. The application team asked us why, quite reasonably, and the answer took three weeks to assemble because nobody in my department had a list of what we install on a machine.
We built one. Across two hundred and fourteen servers, the average was eleven pieces of software that exist only to manage the server rather than to do anything the company sells. Backup. Monitoring. Endpoint detection. An inventory agent. A patching agent. A log shipper. Configuration management. A licence metering agent a vendor requires. Two remote access tools, one of which nobody could account for. And on thirty one machines, the monitoring agent of a product we retired in 2021, still running, still trying to report to a server that has not existed for four years.
Two of them were doing the same job and we were paying for both. Four were on versions the supplier no longer supports. Every single one runs with the highest privilege the operating system offers, reaches out to the internet on a schedule, and updates itself without asking, and not one of them had ever been through the review we apply to an ordinary business application, because installing the agent is treated as part of buying the product rather than as a decision of its own.
The cost is not mainly the licences, though we recovered about thirty thousand pounds a year. It is that our servers are running a quantity of privileged third party code that grew one sensible purchase at a time, and that we could not answer a simple question from our auditor about which suppliers can execute instructions inside our estate.
We now hold a baseline: seven things may be installed on a server, each at a named supported version. Anything new goes through the same assessment as an application, including what it can reach and whether it can be stopped from updating itself. Where a supplier allows it, automatic updates are off and an update is a change like any other. We removed four agents and two consoles, and a purchase now has to state what it installs before it is approved.
The estate is not the servers. It is everything we agreed to run on all of them at once.
– Serguey Shinder
