The NIS2 implementation act is now in force in Germany, and for affected grid operators and utilities (Stadtwerke) the consequences are concrete. They must be able to handle security incidents, they need plans for operations, recovery and crisis management, and they must report significant security incidents to the BSI (the German Federal Office for Information Security) within 24 hours at the latest. This article is an English adaptation of my German original.
Whether an OT incident response plan holds up in a real incident only shows in an exercise. That exercise needs grid control and protection engineering at the table, along with IT and executive management. Grid operators should run a scenario at least once a year, ideally quarterly as a tabletop, in which the control system itself can no longer be trusted. The manual operation at the substation has to be genuinely tried out. After every exercise, the plan gets rewritten.
I'm Max Gilg, an OT cybersecurity expert from Rosenheim. In this series I go through the five SANS ICS Critical Controls one by one, each for a specific industry. This time it's CC1, ICS Incident Response, for power and grid operators in Germany. All incidents mentioned are publicly documented.
Why a plan in the cabinet is not enough
Many organizations have an emergency manual that was approved and ticked off in an audit. It gets interesting at 2 a.m.: the control room sees switching operations nobody triggered. Someone has to decide whether the telecontrol link to the substations gets cut. Someone has to drive out and switch on site, and someone has to report to the BSI. Who does what must be settled beforehand.
That attackers deliberately target power grids is well documented. On 23 December 2015, about 225,000 customers lost power at three Ukrainian distribution system operators. According to the US agency CISA, the attackers rendered systems unusable with KillDisk and destroyed the firmware of serial-to-Ethernet converters in substations (CISA). In April 2022, ESET and CERT-UA analyzed an attack attempt on high-voltage substations of a Ukrainian utility. There, the Industroyer2 malware was to be deployed, alongside several wipers such as CaddyWiper (ESET WeLiveSecurity, 12 April 2022).
A key CISA recommendation from 2016: develop and practice contingency plans that allow safe operation or shutdown even when the ICS is actively working against the process. That is an uncomfortable thought for a control room. Many emergency plans assume the control system keeps working and only a server fails. It looks different when the screens show values nobody can believe anymore. Whether your own team can keep grid operations running for a while without telecontrol is something no document answers. Only an exercise does.
Germany also offers lessons, although not classic cyberattacks on grid operators. According to press reports, after the disruption of the KA-SAT satellite network in February 2022, remote monitoring failed at a large number of wind turbines in Germany. The turbines kept running in automatic mode but could no longer be controlled remotely (pv magazine, 1 March 2022). The remote-control disruption then continued for some time (it-daily, Modern Power Systems). If your telecontrol depends on a single communication path, you should play through exactly this case once.
In Berlin, according to press reports, an arson attack cut parts of a district off the grid, affecting numerous households (SRF, Der Standard). That was sabotage, not a cyberattack. Still, it is instructive for your plan: from restart and recovery to communication with authorities and the public to shift planning during longer outages, these are topics missing from many IT contingency plans.
What NIS2 requires from power and grid operators in Germany
The act implementing the NIS2 directive has been promulgated in the Federal Law Gazette (BGBl.) and is in force. The minimum measures under § 30 (2) BSIG include handling security incidents, maintaining operations with backup management, recovery after an emergency, and crisis management. In addition there are procedures to assess the effectiveness of these measures (§ 30 BSIG). The BSI explicitly counts testing of contingency plans as part of Business Continuity Management (BCM) and points to BSI Standard 200-4 as the method (BSI, NIS-2 BCM, BSI Standard 200-4). That is no coincidence: how effective a plan is that nobody has ever tested is hard to assess.
For reporting, § 32 BSIG prescribes a three-stage process: an initial report no later than 24 hours after becoming aware of a significant security incident, a follow-up report no later than 72 hours after becoming aware of it, and a final report no later than one month after the 72-hour report (§ 32 BSIG). The BSI provides further information on the reporting obligation (BSI, NIS-2 reporting obligation).
Grid operators also deal with the Bundesnetzagentur (Federal Network Agency). It has published drafts intended to merge the existing IT security catalogs for networks and energy installations. General measures for cybersecurity and business continuity (BCM) are to be separated from the specific requirements (Bundesnetzagentur, press release). Until a new catalog is published, the existing catalogs under the EnWG remain the reference, and attack detection systems (SzA) must still be operated (Bundesnetzagentur, Bundesnetzagentur, IT security). Operators should check the final status of the new catalog directly with the Bundesnetzagentur, since at the time of writing it could not be confirmed as adopted. Note that an SzA initially only tells you that something is wrong. How the control room responds must be written in the plan and practiced.
CC1 ICS Incident Response in the control room
In their 2022 SANS whitepaper, Robert M. Lee and Tim Conway put ICS Incident Response first as Critical Control 1. They mean a plan that emerges from operations, focuses on the integrity and recovery of systems, and is practiced with scenarios that fit your own ICS environment (SANS, copy). A post on the SANS blog also recommends running technically grounded ICS tabletops regularly (SANS Blog).
For a grid operator this means: the plan starts with the question of how supply stays secure. Which server gets rebuilt comes later. IT contributes to the plan, but grid control decides when to re-enable systems or switch to manual operation.
Practicing without endangering the grid
Four weeks are enough for the first exercise.
Week 1: Choose a scenario. The scenario should be technically possible in your own installation. For example: the telecontrol link to several substations drops, the operator workstations in the control room are encrypted, or a service provider reports misuse of its remote maintenance access. Or a combination: ransomware encrypts the office IT, an engineering workstation in the control room shows suspicious connections, and unexpected commands appear on an IEC 104 link. That ties IT and OT together, and this is the interface where things usually go wrong.
Week 2: Clarify roles. Define who may cut the telecontrol connection and who decides on the switch to manual operation. Just as important: who has the phone numbers of protection engineering, the grid control system vendor and the on-call duty, on paper. Invite switching supervisors, protection engineering, IT, communications and one person from executive management. CISA provides a free ICS exercise package (Situation Manual) that adapts well to German conditions (CISA ICS CTEP, overview).
Week 3: Play it through. Two to three hours at the table, with a clock. Every participant says what they do at minute 5, at hour 1 and at hour 24. In between come injects, such as "The control room loses visibility of three substations" or "The vendor wants remote maintenance access to the grid control system." The reporting chain is rehearsed too: someone must decide whether an incident is significant, someone writes the 24-hour report, and on weekends that person needs a deputy. Often it only becomes apparent here, for example, that nobody has access to the reporting portal, or that the phone list exists only in the encrypted mail system.
In parallel, it gets practical. A team drives to a substation and checks whether switching on site is possible, who is authorized to do it, and whether the keys are where the plan claims. This works in normal operation, without switching a single device. Recovery also belongs on the test bench: configurations of protection devices, telecontrol components and operator workstations should be available offline, and firmware versions documented.
Week 4: Rewrite the plan. For every gap found, assign an owner and a deadline by which it should be closed. At the next run, check whether that worked. This is exactly one building block for the effectiveness assessment under § 30 BSIG.
Typical gaps in control systems
The following points are typical examples, not study results. A typical gap would be a missing up-to-date asset inventory of the telecontrol substations. In an emergency nobody knows which firmware runs where, and rebuilding takes much longer because first you have to search. Visibility in the OT network can also be missing.
Takeaways
- Run a tabletop at least yearly, better quarterly, with a scenario in which the control system itself can't be trusted.
- Put grid control, protection engineering, IT and executive management at the same table.
- Actually try out manual operation at the substation (possible in normal operation without switching any device).
- Keep phone numbers on paper, and keep device configurations offline with documented firmware versions.
- Rehearse the 24-hour report with a deputy for weekends.
- Rewrite the plan after every exercise, with an owner and a deadline per gap.
If you need support in an actual incident, see OT-Notfallhilfe bei Cyberangriffen (Incident Response).
Original (German): https://ot-cyber.de/blog/notfallplan-im-netzbetrieb-geuebt-statt-abgeheftet.html
Max Gilg is an OT cybersecurity consultant based in Rosenheim, Germany (OT-Cyber.de).
