NGFW High Availability: Active-Passive, Active-Active and Synchronization

Does failover really work in a two-firewall HA cluster? Active-passive and active-active, session synchronization, split-brain, failover behavior during a planned upgrade and failover testing.
Technical summary
Technology
NGFW HA, FortiGate FGCP, Palo Alto HA, VRRP, LACP, session synchronization
Topic
Failover behavior, synchronization and testing in firewall clusters
Level
Intermediate–advanced
Prerequisites
  • Firewall interface, zone and routing configuration
  • Basic knowledge of LACP/MLAG on the switch side
  • A maintenance window and change management process
In this article8 sections

Putting in two firewalls and setting up an HA cluster doesn’t mean failover will work. The firewall is the single transit point for internet egress, inter-branch VPN and often routing between internal zones; in a single-device setup, a hardware failure, firmware bug or maintenance is a direct business outage. High availability (HA) reduces this risk, but only if synchronization, heartbeat and monitoring are set up correctly. Below we explain, vendor-neutrally, the real behavior of HA modes, the limits of synchronization, split-brain risk and failover testing; the examples use FortiGate and Palo Alto Networks terminology.

Why firewall HA is a separate design topic

In router or switch redundancy, two devices working independently is often enough; the routing protocol moves traffic onto the healthy path. A firewall, however, is a stateful device: there’s a record of every TCP connection, NAT mapping, IPsec SA and identity mapping. If traffic goes to another device unaware of this record, the device treats the packet as belonging to an unknown session and drops it. No record, no passage. That’s why firewall HA in an enterprise cybersecurity infrastructure has to answer three questions together: which device carries the traffic, how state information is copied, and how the devices verify each other’s state.

HA covers not only the inside of the firewall but also its surroundings. Two firewalls connected to a single core switch, a single internet line or a single power source lose the redundancy the cluster provides to that single point of failure in their surroundings. This is the common picture: two firewalls, two cables, one switch. For redundancy options on the switch side, the switch stack, MLAG and vPC comparison on the network site shows the way.

Active-passive versus active-active

In active-passive mode, one device carries all the traffic while the other waits, receiving state information. In active-active mode, both devices process traffic; but how this is done varies significantly by vendor. In FortiGate FGCP active-active mode, the primary device receives the traffic and distributes the security inspection of some sessions to the secondary device. Palo Alto Networks, on the other hand, positions active-active mode primarily for special topologies such as asymmetric routing, and in this mode requires an additional HA link (HA3) for packet forwarding.

Criterion Active-passive Active-active
Configuration complexity Low High
Troubleshooting Traffic on a single device You have to track which device a session is on
Asymmetric traffic Generally not supported May be supported, depending on the vendor
Capacity after a failure Unchanged (single device) Drops to a single device
Suitable scenario Most enterprise edges and data centers Special topologies and load distribution needs

There’s one rule on capacity: in an active-active cluster, the total load shouldn’t exceed what a single device can carry. If it does, the cluster stays up during a failure but can’t carry the traffic. When this rule is applied, in most organizations the benefit of active-active mode is limited while its complexity is clear; during a failure, the question “which device was the session on?” lengthens troubleshooting. Device sizing criteria are in the enterprise firewall selection guide.

Session, configuration and state synchronization

Think about synchronization in three layers:

  • Configuration: policy, object, VPN and system settings are copied from the primary device to the secondary. Device-specific settings such as interface IPs, HA priority and the management interface are generally not synchronized. Making changes only on the primary device should be an operating rule.
  • Session table: lets open TCP connections continue after failover. On FortiGate, session synchronization is off by default and is enabled with session-pickup; connectionless sessions such as UDP and ICMP additionally require session-pickup-connectionless.
  • Other state information: IPsec SAs, DHCP leases, user-IP mappings, the routing table. Which of these are synchronized varies by vendor and version.

In most products, sessions undergoing proxy-based inspection and deep SSL inspection aren’t preserved on failover; they’re re-established. On the user side this is usually limited to reloading a page, but long file transfers and some application connections may drop. Verify which session types are preserved in the HA documentation for the version you use (Fortinet, Palo Alto Networks).

An example FortiOS active-passive configuration:

config system ha
    set group-name "FW-HA"
    set mode a-p
    set password <strong-password>
    set hbdev "port9" 50 "port10" 50
    set session-pickup enable
    set session-pickup-connectionless enable
    set override disable
    set priority 200
    set monitor "port1" "port2"
end

Here two heartbeat interfaces (port9, port10) are defined, session synchronization is enabled, and the link state of port1 and port2 is monitored. override disable, the default value in FortiOS, prevents a device returning from a failure from triggering a second failover on its own to become primary again. Failing back becomes a planned operation. And there’s no second outage.

Devices learn that each other is alive through heartbeat packets. If all heartbeat paths are cut, both devices assume the other has died and become active. This is called split-brain: the same virtual IP and MAC addresses are advertised from two devices, the switches’ MAC table flaps between two ports, sessions drop erratically and configurations start to diverge. The symptom is confusing too. The network works “sometimes.”

Countermeasures:

  • Use at least two physical heartbeat links; if possible on different interface modules and with direct cables.
  • On Palo Alto Networks, define a backup link (HA1 backup) for the control link (HA1) or a heartbeat backup over the management interface.
  • If you have to carry heartbeat traffic over a switch, use a separate VLAN dedicated to this purpose; don’t share this VLAN with other traffic.
  • Don’t set heartbeat timers below the vendor’s recommended values; they produce false failovers at moments of high CPU.
  • Send HA state change events to the SIEM and define a separate alarm for the situation where both devices appear active at the same time.

Failover triggers: interface and path monitoring

Even while the device is running, it may be unable to carry traffic: the WAN interface may go down, or access to the service provider’s network may be lost even though the link is up. That’s why two levels of monitoring are defined:

  1. Interface monitoring: the physical link state of critical interfaces. When an interface goes down, the cluster priority changes and the healthy device becomes active.
  2. Path monitoring: a ping or similar probe to a specific target (the ISP gateway, a router in the data center). Link monitor on FortiGate and path monitoring on Palo Alto Networks are used for this purpose.

In path monitoring, choosing the target is the job itself. Monitoring a single public IP on the internet leads to unnecessary failover during that target’s own outage. Define multiple targets and use the “when all fail” condition. Dual lines and line redundancy are a WAN design topic; HA’s job is to handle device and interface failures.

How quickly failover is reflected for the user also depends on how fast the switch updates its MAC table. Vendors send gratuitous ARP at failover; in setups using a virtual MAC, test that these ARPs reach the switches and that port security or dynamic ARP inspection doesn’t block the update. If the ARP update doesn’t reach the switch, the firewall fails over within seconds, but the switch keeps sending traffic to the old port until the MAC entry ages out; and the problem gets looked for in the firewall by mistake.

Failover behavior during a planned upgrade

One of the common uses of an HA cluster is planned maintenance. The flow is: the passive device is upgraded first, then a controlled failover is done, then the former active device is upgraded. In FortiOS, the uninterruptible upgrade option applies this sequence automatically; the setting’s name and default vary by version. On Palo Alto Networks, the process is run manually by suspending the passive device. Which sessions and NAT mappings are preserved during the transition depends on the device, version and session type; sessions passing through deep TLS inspection and VPN tunnels should be checked separately. Having two devices doesn’t mean all application sessions will be preserved.

  • Before you start, take a configuration backup and check the vendor-supported upgrade path; some versions require an intermediate version.
  • Devices in the cluster running different versions is a temporary state acceptable only for the duration of the upgrade. Keep this time short.
  • After the upgrade, verify that the configuration checksums (diagnose sys ha checksum cluster on FortiGate) match on both devices.

For basic hardening steps in a single-device setup, see the FortiGate setup checks article.

Failover test checklist

Run the test in a maintenance window, with a rollback plan and observation tools ready:

  • Were the HA state, checksums and number of synchronized sessions recorded before the test (get system ha status on FortiGate)?
  • Is a long-lived test connection (an SSH session, a continuous ping, a VoIP call) being watched throughout the failover?
  • Were manual failover, power loss on the active device, pulling the cable of a monitored interface and making the path monitoring target unreachable each tried separately?
  • Do IPsec tunnels, remote access VPN users and NAT-published services work after failover?
  • When the failed device came back, was there an unwanted second failover?
  • Did HA state change logs and alarms appear in the SIEM as expected (see firewall log analysis)?
  • Were the measured outage duration and the types of sessions dropped written in the test report?

Trying only a manual failover isn’t enough. Real failures come from the power cable. If you’d like the design and test results of an existing cluster assessed with an independent eye, we handle this within firewall installation and rule management.

Frequently asked questions

Does an active-active setup double firewall capacity?

No. In normal operation the load can be spread across two devices, but when one device fails, all the load falls on the remaining one. For the cluster to survive a failure, total traffic shouldn’t exceed the capacity of a single device; capacity planning should be done on a single device, not on double.

Technically it can. The preferred method is to connect the two devices directly with a cable and use at least two independent heartbeat links. A single heartbeat passing through a switch can, on a switch failure, cause both devices to think they’re active, i.e. a split-brain.

Why do existing sessions drop after failover?

The most common reasons: session synchronization being off, synchronization for connectionless protocols not being enabled separately as required, and proxy-based inspection sessions not being synchronized. Switches updating their MAC table late also causes a brief loss of access after failover.

How often should failover testing be done?

Always after installation, after every major version upgrade and after topology changes; beyond those, at least once a year in a planned maintenance window. Don’t assume an untested HA setup will behave as expected in a real failure.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Cybersecurity

Firewall, VPN, endpoint security and threat prevention. We reduce the attack surface and measure and report the risks.

Knowledge Center

Other articles