OT Incident Response Plan: Safely Bringing Production Under Control

Who makes the isolation decision during a security incident in OT? A decision authority matrix, an isolation decision tree, PLC and HMI evidence collection, verified restore and exercises.
Technical summary
Technology
PLC, HMI, SCADA, historian, NIST SP 800-61, NIST SP 800-82
Topic
The cyber incident response process in an OT environment
Level
Intermediate–advanced
Prerequisites
  • An existing IT incident response plan
  • An OT asset inventory and network drawing
  • Participation of the operations and maintenance team
In this article8 sections

When a cyber incident starts in a factory, the first question shouldn’t be “which system was affected” but “is the process safe.” Incident response plans written for IT don’t ask this question. Reflex steps such as disconnecting a server from the network, locking an account or rebuilding a machine can directly affect production, equipment and human safety in OT. In this guide we explain how we write the OT section to add to an existing IT plan: roles, the isolation decision, evidence collection, recovery and exercises.

The differences between IT and OT incident response

Topic IT approach What to watch for in OT
Priority Data confidentiality and integrity Human and environmental safety, then process continuity
Isolation Disconnect the suspicious system from the network immediately The effect of disconnection on the process is assessed first
Shutdown The system is shut down if needed If the HMI shuts down, the operator can’t see the process; a controlled shutdown may be needed
Evidence Disk and memory image PLC program, diagnostic buffer, process data; a memory image isn’t possible on every device
Recovery Rebuild from an image Verified project file, firmware and a commissioning test
Decision owner IT / security team A joint decision with the production and process safety manager

The common consequence of these differences is simple: the OT plan isn’t a document the security team can apply on its own. The production manager, maintenance chief, automation engineer and the OHS/process safety officer should take part in writing it. If production isn’t at the table, the plan stays on paper.

Roles and decision authority: production, maintenance, security

What wastes the most time during an incident isn’t a technical problem but uncertainty about who can make which decision. Phone calls go around, and everyone waits for someone else’s approval. The matrix below is a starting template (K: decides, U: implements, D: consulted, B: informed):

Decision Security Production Maintenance/Automation
Declaring the incident and its severity K D D
Cutting the IT/OT connection U K D
Stopping the line or switching to manual mode D K U
Restoring the PLC program D B K / U
Evidence collection K / U B U (PLC/HMI side)
Approval to return to production D K D

Be sure to clarify two points in the matrix. The first is the night and weekend shift: who makes these decisions at that hour, and who are their deputies? Names and phone numbers should be written in the plan. The second is the role of the machine builder (OEM) and system integrator: in which situations they’ll be called, what access they’ll work with, and whether they’ll be allowed to connect remotely during an incident. If this last question is debated during the incident, hours are lost.

Detection and initial assessment

An OT incident arrives through different doors: an alarm from the OT monitoring platform, a ransomware incident that started in IT, an operator’s report that “the HMI is changing values by itself”, or an unexplained process fault. However well OT attack detection is set up, the first signal is likely to come from the floor. How the operator reports a suspicious situation, and to whom, should be defined in the plan; otherwise the report goes to maintenance as a fault and stays there.

The questions to answer in the initial assessment are fixed with a short checklist:

  • Is the process in a safe state right now? Are the safety systems (SIS) working normally?
  • Can the operator see and control the process (is the HMI display reliable)?
  • Did the incident originate in IT, or start within OT? Which zones and cells are affected?
  • Has there been an approved change, maintenance or remote access session in the last 24 hours?
  • Is the incident spreading (newly affected HMIs, servers or cells)?

An answer of “the operator can’t see the process” or “the values on the HMI can’t be trusted” takes the incident straight to the highest severity. From that moment on, it stops being a cybersecurity question and moves to process safety procedures.

Isolation decision tree: cutting the network, switching to manual mode

Isolation options are ordered from least to most disruptive, and what each one will do to which line is known in advance. A simplified decision tree:

  1. Is there a process safety risk? If so, the process safety procedure kicks in: a controlled shutdown or bringing it to a safe state. Cyber steps come after this.
  2. Is the incident only on the IT side? If so, the connection at the IT/OT boundary is cut and OT runs in “island mode.” The prerequisite is knowing in advance how long production can be sustained without IT services; this dependency analysis is covered in the how ransomware spreads to OT guide.
  3. Is the incident in the OT DMZ or level 3 systems? Systems such as the historian, remote access server or patch server are separated from OT; HMI and PLC traffic continues.
  4. Is the incident on specific HMIs or engineering workstations? If the cell has a backup HMI or a local operator panel, the affected station is disconnected from the network. If not, the line is put into manual mode and operator control shifts to the floor.
  5. Have the PLC program or setpoints been changed? The line is stopped, PLC diagnostic information and the current program are taken, and the verified program is restored.

At each step, “who decides” is read from the matrix and “how it’s done” from the technical annexes. Which switch port or firewall rule to shut isn’t looked up during the incident; it should be written in advance and tried during a maintenance shutdown. The common gap: the decision to cut comes quickly, but nobody knows which cable or rule to cut.

Evidence collection and recording in OT

In OT, evidence collection races against recovery. Production wants to restart as soon as possible; yet the moment the PLC program is restored, the modified program is lost. A minimum evidence set should be taken before restoring:

  • PLC: An online backup (upload) of the current program, the CPU diagnostic buffer (diagnostic buffer on Siemens), the mode and operating state, the firmware version. The program taken is compared with the verified reference project file.
  • HMI and engineering workstation: On Windows-based systems, memory and disk images, event logs, the HMI application’s alarm and audit records.
  • Network: The OT sensor’s packet or session records from before and during the incident, firewall and switch logs.
  • Process data: The relevant tags in the historian; a timeline of setpoint and output changes.

Clock synchronization is often weak on OT devices; PLCs never connected to NTP are not rare. Next to each piece of evidence, note the difference between the device’s time at that moment and the reference time, or the timeline can’t be built. Taking hashes of the evidence and recording who took it and when is the same as chain-of-custody practice in IT; details are in the forensic evidence preservation after a cyber incident guide.

Recovery: restoring the PLC program and HMI

The preparation before the incident determines the speed of recovery. If during the incident people are searching for “which is the last working project file”, recovery stretches to hours, sometimes days. There are three different versions on the engineering laptop, a USB stick and the OEM’s email; nobody can say which one matches what’s on site. Preparation covers:

  • For each PLC, a verified, versioned project file stored in a location independent of the OT network, and its hash.
  • Current system images for HMI, SCADA and engineering workstations, license files and knowledge of how license keys are moved.
  • Firmware files and the vendor’s restore procedure.
  • A clean engineering workstation: it must be certain that the computer that will connect to the PLC during recovery hasn’t been affected.

After the restore, the line isn’t put straight into production. The automation team carries out commissioning that includes I/O checks, testing of safety functions and a short trial run; the approval to return to production is given by the decision owner in the matrix. An untested restore is a second incident. For the general framework of recovery preparation on the IT side, the recovery readiness against ransomware guide complements this section.

Plan validation and exercises

A written plan becomes reliable only when it’s tested with an exercise. The order we recommend for OT:

  1. Tabletop exercise: The decision flow is talked through over a scenario with production, maintenance, security and management. The goal is to find gaps in decision authority and communication.
  2. Technical validation: During a maintenance shutdown, the step of cutting the IT/OT connection is actually carried out; which services production depends on is observed. The license server, time server and file shares often produce surprises in this test.
  3. Restore test: In a test environment or a maintenance window, a PLC program and an HMI image are restored from backup; the time is measured.

The output of each exercise should be concrete: an updated contact list, a corrected decision matrix, backups found to be missing. An exercise with no output is a meeting. For the plan structure, NIST’s incident response guide SP 800-61 Rev. 3 is the reference, and SP 800-82 Rev. 3 for OT-specific considerations. To prepare the OT section of the plan and run the first exercise, the incident response readiness and exercises service can be used, and for technical preparation on the OT side, the OT/ICS security service.

Frequently asked questions

Is the existing IT incident response plan enough for OT?

Usually not. The IT plan treats isolation and shutdown as default steps; in OT these steps can affect process safety. The skeleton of the IT plan is kept, and OT-specific roles, an isolation decision tree and recovery annexes are written on top of it.

Who should decide to cut the IT/OT connection during an incident?

The security or network team technically carries out the cut; because of its impact on production, it makes the decision together with the production manager. This authority and its deputies should be written in the plan in advance.

Why can restoring the PLC program right away be a problem?

A restore wipes out the modified program and diagnostic information. First a backup of the current program and the diagnostic buffer are taken and compared with the verified reference program; the restore is done after that.

How often should an OT incident response exercise be done?

There’s no fixed rule. The common approach is a tabletop exercise at least once a year and a plan update after major organizational or infrastructure changes. Technical restore tests can be tied to planned maintenance shutdowns. Handle the technical response steps together with the cyber incident response and notification plan, which includes decision owners and notification processes.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Cybersecurity

Firewall, VPN, endpoint security and threat prevention. We reduce the attack surface and measure and report the risks.

Knowledge Center

Other articles