If you connect a server restored from backup after a ransomware attack directly to the production network, you may also bring back the back door the attacker left behind. And their still-open access. An isolated recovery environment (clean room) is an environment where restored systems are brought up and examined on a network separate from production, with separate identities, and moved into production in a controlled way only after they’re verified clean. Below we explain the network, identity, tooling and capacity design of this environment and the order of verification inside it.
Why you don’t restore directly into production
In a ransomware incident, encryption is the final act. Before it, the attacker has got into the network, escalated privileges, set up persistence and targeted the backups. That makes restoring risky in three ways:
- The persistence mechanism may be in the backup: A service, scheduled task or web shell the attacker set up may also be present in backups taken weeks before encryption.
- The attacker may still be in the environment: If compromised accounts, VPN access or the command-and-control connection haven’t been closed, the restored system immediately becomes a target again.
- The initial access gap may not be closed: A system brought back without patching the vulnerability where the attack started is taken over again by the same route.
The isolated environment is a waypoint for resolving these three uncertainties without putting production at risk. Which backup to use is a separate decision; we cover it in the choosing a clean recovery point article. The topic here is the place where that backup can be safely opened.
Components of the isolated environment
Trying to build the environment from scratch during an incident drags recovery out for days. Nobody wants to be running cables that night. That’s why, in an enterprise cybersecurity plan, the components need to be ready, documented and tested in advance.
| Component | Purpose | Design note |
|---|---|---|
| Compute resources | Running the restored systems | Hosts or a cloud subscription managed separately from production |
| Storage | Restore and instant recovery | The backup repository’s read speed should be tested for instant recovery |
| Network | Connecting systems to each other and to tools | No routing to production; egress off by default |
| Identity | Managing the environment | Local or separate accounts independent of production AD, MFA |
| Infrastructure services | DNS, NTP, update repository | Inside the environment, independent of production |
| Toolset | Investigation and scanning | EDR, scanners, forensic tools; kept current with offline signatures |
| Jump point | The team’s access to the environment | A clean workstation; not a member of the production domain |
The most sensitive component is identity. Managing the isolated environment with production Active Directory means carrying compromised identities into this environment too. The environment should have its own management accounts, MFA and password vault. The same logic applies to backup infrastructure; details are in the identity isolation in backup infrastructure article.
Network isolation and controlled egress
Setting up the isolated environment’s network as three zones simplifies the work:
- Quarantine zone: Where restored systems are first brought up. Systems talk to each other and to tool servers; there’s no egress at all.
- Tools zone: Scanning, the EDR management server, the log collector and the jump point. It can reach quarantine; quarantine reaches this zone only through defined ports.
- Verified zone: Where systems that pass inspection are moved for application testing. If needed, it can have limited, logged egress.
If internet egress is needed (for security patches or EDR signature updates), open it through an allow-list-based proxy or firewall, only to specific destinations, with all traffic logged. Logging DNS queries from quarantined systems is the cheapest way to catch malware still trying to reach a command-and-control server. You can configure the DNS server inside the environment to log queries for unknown domains without resolving them. These logs are often the first place to look in the console.
Restored systems carry the same IP addresses and machine names as in production. That’s why having no routing between the isolated environment and production is essential not only for security but also to prevent address conflicts. If access from inside is needed, address translation is used; Veeam’s virtual lab solves this problem with a proxy appliance and masquerade addresses.
On-premises, cloud and hybrid options
| Option | Strength | Point to watch |
|---|---|---|
| Separate on-premises hosts | Close to the backup repository; fast restore and instant recovery | Cost of idle hardware; must really be managed separately from production |
| Cloud subscription | Pay as you use; physical separation from production | Data transfer time, egress cost; a separate account and identity are mandatory |
| Hybrid | Identity and critical core on-premises, scale in the cloud | Both environments must be exercised together; complexity increases |
A mistake we often see with the on-premises option is setting aside a few hosts from the production cluster and managing them with the same vCenter or the same Hyper-V management infrastructure. If the production management plane has fallen, the isolated environment has fallen too. Separate hosts should be set up with a separate management interface, separate identity and, if possible, separate network hardware. If the cost of idle capacity is unacceptable, these hosts can be used for testing or development; but they should be planned so they can be cleared within minutes during an incident.
With the cloud option, recovery time is determined by whether the backups have already been copied to the cloud. If backups that sit only on-premises are to be uploaded to the cloud during an incident, transfer time depends directly on bandwidth. Don’t commit to an RTO without measuring this time in an exercise. An unmeasured RTO is a guess.
Verification steps in the environment
Each restored system is examined in the order below. For critical systems, all steps can be applied; for low-priority systems, a risk-based shortened version.
- First boot without a network: The system is first started with its network adapter disconnected, and boot and service states are checked.
- Malware scanning: A full scan is run with EDR or antivirus with current signatures; a search is run with the IOC list derived from the incident (file hashes, file names, domain names).
- Persistence check: Services, scheduled tasks, autostart registry keys, WMI event subscriptions, startup folders, local administrator group members, and on Linux, cron and systemd units are examined.
- Account and privilege check: Accounts created or whose privileges were changed during the incident timeline are identified.
- Closing vulnerabilities: The gap used for initial access and critical security updates are applied.
- Renewing credentials: Service accounts, local administrator passwords and application secrets on the system are changed.
- Application testing: Together with a business unit representative, it’s verified that the application works functionally.
- Record: The recovery point used, the checks performed and their results are kept in a system-specific record.
On Windows, you can quickly collect part of the persistence check with PowerShell. The commands below produce a starting list; compare the output with a known-good baseline configuration, because reading it on its own is misleading:
Get-ScheduledTask | Where-Object State -ne 'Disabled' |
Select-Object TaskPath, TaskName, @{n='Action';e={$_.Actions.Execute}}
Get-CimInstance Win32_Service |
Select-Object Name, StartMode, StartName, PathName
Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Run',
'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce'
Get-CimInstance -Namespace root\subscription -ClassName __EventConsumer
Get-CimInstance -Namespace root\subscription -ClassName __FilterToConsumerBinding
# Local Administrators group (by SID; the group name differs on localized systems)
Get-LocalGroupMember -SID 'S-1-5-32-544'
Moving clean systems into production
Moving a system verified in the isolated environment into production relies on the assumption that the production network has also been cleaned. A move made without verifying this assumption throws away all the assurance the isolated environment gave. Before moving into production, meet these conditions:
- The attacker’s known access paths (VPN accounts, remote access tools, exposed services) have been closed.
- Privileged account passwords have been changed, the KRBTGT password has been reset twice after replication completed, and the identity infrastructure has been made trustworthy again.
- The target network segment has been created with new rules; compromised systems can’t reach this segment.
- EDR and central log collection are on for the moved systems from the very first moment.
- An increased monitoring period and a responsible person are defined for the moved systems.
The move is done in waves: first identity and infrastructure services, then the applications that depend on them. Leaving a short observation period after each wave lets you catch signs of reinfection early. Don’t tear down the isolated environment until the incident is closed. If reinfection happens, the same process will be run from the start.
The isolated environment’s design is only validated through an exercise; without measuring capacity, transfer time and the real durations of the verification steps, RTO estimates remain assumptions. The environment’s design and exercise can be handled together within cyber incident response readiness and exercises, and the backup-side components within backup security and ransomware readiness. The primary source for the general framework of the recovery process is NIST’s SP 800-184 guide for cybersecurity event recovery.
Frequently asked questions
Is an isolated recovery environment the same as a SureBackup virtual lab?
Their logic is similar; their purposes differ. A virtual lab sets up an isolated network managed by the backup server to routinely test that backups can be opened. An isolated recovery environment after a cyber incident, on the other hand, is designed with separate identity, separate tools and controlled egress, on the assumption that the backup infrastructure may also have been compromised. A virtual lab can be used as a component of this environment. To tie the isolated copy to a working recovery scenario, define the test and verification steps in the design of your backup and disaster recovery infrastructure.
How much capacity should be set aside for the isolated environment?
Not enough for all of production, but enough to carry the systems that will come up in the first wave. This usually includes identity infrastructure, DNS, management tools and a few of the most critical business applications. Calculate CPU, memory and storage needs from the inventory of these systems; if instant recovery will be used, test the backup repository’s read performance separately.
Is it safe to set up an isolated recovery environment in the cloud?
If designed correctly, yes. The environment should be set up in an account or subscription separate from the production cloud subscription and the production identity provider, network connectivity should be off by default, and management access should be done with separate, MFA-protected accounts. Measure the data transfer time and egress costs in an exercise.
Can a system verified as clean be connected directly to the production network?
It shouldn’t be connected before the production network is cleaned. Otherwise a still-active attacker or persistence mechanism can take over the restored system again. First a clean network segment and new credentials are prepared, and systems are moved into this segment in waves under increased monitoring.