Skip to content

OPNsense HA Updates (CARP Pairs) ​

Since v3.7.0 DATAZONE Control executes OPNsense updates as update runs. A run is a state machine in the backend that updates several firewalls in the right order, verifies every step against hard facts and decides on its own what to do about every known disturbance: retry, abort cleanly or heal the state. The goal is the unattended night run - without a technician having to rescue a half-finished job in the morning.

Who is this for?

This page describes the behaviour in detail. If you just want a schedule: OPNsense > Jobs > New, type Update + Reboot, HA mode Rolling (default), done.

Update Types ​

TypeWhat happensReboot
System UpdatePackages only (opnsense-update -pt). Base/kernel updates are not installed but reported as reboot pendingnever
Update + RebootPackages + base/kernel; reboot only when OPNsense really requires it (base/kernel installed or system.firmware.reboot)when required
Major UpgradeMajor releases step by step up to the target or the latest offered release (e.g. 23.7 → 24.1 → … → 26.7), with a regular update before each steptwo reboots per step

All three types use the same machine; the only difference is what the agent is allowed to do.

Prerequisites ​

  • Agent ≥ 3.7.0 on every firewall. Older agents are always skipped in a run (reason agent too old) - the old agent reboots unconditionally and knows nothing about CARP.
  • For rolling updates: both nodes online, CARP roles exactly one MASTER and one BACKUP, no active maintenance mode, peer assignment known (see below).
  • pfsync and configuration synchronisation set up as recommended by OPNsense.

Column "Update": is the firewall ready? (since v3.12.0) ​

The OPNsense list shows a dot per firewall in the column Update:

  • green "ready": an update with reboot would touch this firewall now.
  • red with reason: the run would skip it. Hovering shows what to do.
  • blue "Update in progress": the firewall is part of a running run.

The column uses the same rules as the update run. Whatever is red there is skipped by the run with the same sentence. The same sentence appears in the run detail and in the notification (Teams, e-mail).

ReasonWhat to do
Agent too old (below 3.7.0)Update the agent first.
HA partner's agent too oldUpdate the agent on the partner.
Firewall offline / HA partner offlineCheck the connection. A pair is only rebooted with both nodes up.
CARP maintenance mode activeLeave it in OPNsense under Interfaces > Virtual IPs > Status once nobody is working on it.
HA roles unclearCheck the CARP status of both nodes. There must be exactly one MASTER and one BACKUP.
HA partner not detectedBoth nodes must be in Control. Check HA sync (System > High Availability).
HA pairing disputedAnother firewall claims to be the partner. Assign the real partner as admin (see Mutual confirmation).

Firewalls without CARP VIPs only check agent version and online state. A package-only update without reboot still runs on the reachable node of a pair with an offline partner; the column shows the stricter case with reboot.

There is no job setting for this. Legacy IPsec, legacy rules and MSS are deliberately not checked by the update but by the checks of the same name.

While a firewall is part of an update run, notifications do not report it as offline, until the health check after the step has ended. The planned reboot therefore no longer triggers an offline alert plus all-clear. The result of the run is reported as before.

HA Pair Detection ​

The agent reports the CARP state in its heartbeat (role per VHID, maintenance, demotion, pfsync peer IP from config.xml). Control derives pairs from it:

  1. The pfsync peer IP of a firewall is an interface IP of another firewall of the same customer.
  2. Fallback: identical CARP VIPs (VHID + address) on two firewalls of the same customer.

The assignment is shown in the firewall table (column HA: MASTER / BACKUP / MAINT / INIT, peer in the tooltip) and in the detail panel. Via the API (POST /api/firewalls/<id>/ha-peer) a pairing can be set and locked manually. A locked pairing (locked) is respected everywhere: neither the auto-detection nor the run planning nor the Update column pair a locked firewall by identical VIPs. Only a mutual assignment counts as a pair. Firewalls with CARP VIPs but no known partner are skipped for updates with reboot (HA partner unknown) - never rebooted blindly.

Mutual confirmation (since v3.48.13) ​

The heartbeat data comes from the firewall's own agent. So that a single (e.g. compromised) firewall cannot register itself as the partner of a real pair:

  • A pair is only created automatically when both nodes, each from its own heartbeat, yield exactly the other one as partner (pfsync peer IP of A is an interface IP of B and vice versa, or both report the same CARP VIPs and each finds only the other). One firewall's heartbeat only changes the other firewall's assignment if the other's last heartbeat points back to it.
  • Only firewalls of the same customer with a heartbeat in the last 15 minutes count. Firewalls without a customer are never paired automatically (assign them manually). Pairs across customers are never created, not even manually.
  • If several sites of one customer use the same sync network, the node whose own pfsync peer IP points to this firewall, or which has the same VIPs, wins.
  • An existing mutual pair is never switched automatically. If another firewall shows up with a matching IP or the same VIPs, the pair stays as it is and Control records an HA conflict: log entry, warning in the HA box of the firewall (Updates tab) and the reason HA pairing disputed in the Update column. While the conflict exists, the firewall and its partner are not touched by any update run, and auto-detection for this firewall is paused.
  • To resolve: as admin, assign the real partner (POST /api/firewalls/<id>/ha-peer with peer_id and locked). This clears the conflict of both firewalls. Then check the other firewall - a firewall reporting the IP or VIPs of another pair is misconfigured or tampered with.

Node IP instead of VIP

Since v3.7.0 the agent reports its own interface address, no longer the CARP VIP that is identical on both nodes.

Flow of a Rolling Run ​

Schedule due
   │
   ├─ Planning: build pairs, check preconditions, record old agents / offline nodes as "skipped"
   │
   ├─ Step 1 ── BACKUP node ────────────────────────────────────────────────┐
   │     preflight → backup(config.xml) → packages → basekernel → report  │
   │     → reboot → node is back (new boot_time) → health check            │
   │                                                                        │
   ├─ Settle time (default 300 s, pfsync resync)                            │
   │                                                                        │
   ├─ Step 2 ── MASTER node ────────────────────────────────────────────────┤
   │     preflight → ... → CARP maintenance ON → peer takes over (≤ 90 s)   │
   │     → reboot → node is back → agent releases maintenance itself        │
   │     → health check (exactly one MASTER in the pair, no maintenance)    │
   │                                                                        │
   └─ Result in the Jobs tab ──────────────────────────────────────────────┘

Single firewalls (no HA) run in parallel as one step each. Different HA pairs also run in parallel; inside a pair strictly one after the other.

Health Check After the Reboot ​

A node counts as healthy only when all conditions hold for at least 60 seconds and a further heartbeat arrived in that time (a single green reading is not enough). If the boot time changes again during verification (second reboot of a major upgrade) the clock restarts; if no heartbeat arrives for more than 3 minutes the node does not count as healthy:

  • version equals the target version reported by the agent (major: prefix of the target version)
  • kernel version changed if a base/kernel update was installed
  • no reboot pending
  • HA: maintenance off, CARP role MASTER or BACKUP, peer online, exactly one MASTER in the pair
  • at least one gateway online if one was online before the update

Timing: Health check timeout (rolling at least 900 s, single firewalls 600 s, major 2700 s) for the return after the reboot, then 300 s (major 600 s) for the conditions to converge. If the node is stuck in maintenance, Control sends one remediation task (OPN_CARP_MAINTENANCE off) and doubles the convergence window for it.

Roles after the run

OPNsense has preempt disabled by default. After the run the former BACKUP node may stay MASTER. That is not an error - the requirement is exactly one MASTER.

CARP Maintenance ​

Before rebooting the MASTER the agent enables the persistent CARP maintenance mode (configctl interface carp_set_status maintenance, demotion 240, stored in config.xml). The peer takes over all VIPs; if it does not within 90 seconds the agent reverts the mode and the step fails - without a reboot.

Because the mode survives the reboot the node comes back as BACKUP. The agent releases the mode itself after start-up (once the system has been up for 60 s and configd answers, retried for up to 15 minutes) - regardless of backend reachability. The BACKUP node never gets maintenance (it carries no traffic).

Since v3.14.2 additionally:

  • Maintenance is only enabled when a reachable partner can take over. A MASTER without an online partner (policy update the online node anyway, single target) is rebooted without maintenance - the reboot is a short outage, not a failover. The step carries a note.
  • If configd does not answer, the maintenance state is unknown, not "off": the agent starts no update (error class transient, retried), deletes no release marker and never reports success for the manual action.
  • The role is re-read right before the maintenance decision (after an hour of package installs the peer may have failed over).
  • The agent stays locked until the actual reboot (no agent update, no second task) and releases the maintenance after a restart only when OPNsense has no further reboot pending (second reboot of a major upgrade).

Since v3.48.3:

  • Control allows maintenance for every node of a pair whose partner is online - no longer only for the node that was MASTER at the start. The agent decides right before the reboot on the role it reads then: only a node that is MASTER on at least one VHID gets maintenance. A BACKUP that became MASTER meanwhile, a mixed node and pairs found only via identical VIPs fail over cleanly; a pure BACKUP (also BACKUP + INIT) never gets maintenance. Requires agent 3.48.3 (older agents also toggled mixed nodes without a MASTER VHID).
  • If a step fails after the agent had already enabled maintenance, Control sends the release (OPN_CARP_MAINTENANCE off) at once - before, only a step in the health check got it.

Manual: detail panel > Updates > Enable/leave CARP maintenance.

Schedule Options ​

OptionMeaningDefault
HA modeRolling (backup → master), Skip HA pairs, Parallel (legacy behaviour, full outage during reboot)Rolling
Settle timeWait between the two nodes300 s (major min. 60 s)
If a peer is offlineSkip the pair or Update the online node anyway - then without CARP maintenance, the reboot is a short outageskip
RetriesRepeated attempts on mirror/network errors or an interrupted agent (5/10/15 min apart)3
Health check timeoutWait for the return after the reboot (rolling at least 900 s)900 s
Maintenance windowAfter it ends no further step is started (a running one is still verified); allowed 15-720 min180 min in the UI (API default 60)
Target versionMajor upgrade only: Latest or YY.1 / YY.7 (Community) or YY.4 / YY.10 (Business)Latest

A schedule that targets only one node of a pair does not silently pull in the partner: it updates that node only (with maintenance if it is MASTER and the partner is online). If maintenance is already active on the node it is skipped (maintenance active), not failed. The busy check covers the whole pair: while a step runs on gw1, a second run on gw2 waits and a manual task is rejected with HTTP 409. Since v3.48.3 this also applies to plain tasks: Install + Reboot, reboot, major upgrade, CARP maintenance on or config restore on gw2 are rejected (HTTP 409, naming the busy node) while an update, reboot or restore task is open on gw1; a bulk action skips the second node of a pair. Agent updates and leave CARP maintenance are never blocked.

Before the second node of a rolling pair is touched with reboot permission, Control checks its partner again (v3.48.3): online (heartbeat within 3 minutes, the same rule as the planning), no CARP maintenance, its step not failed. Otherwise the node stays as it is - untouched it is skipped (HA partner offline / CARP maintenance active / partner node not successful), a major chain that already changed it stops on the release it reached (run completed with warnings). This holds between all steps of a major chain too.

What Happens on Disturbances? ​

DisturbanceBehaviour
Mirror/repository unreachablestep is retried (5/10/15 min), then the run fails, pair untouched
Disk too full (less than 1 GB free, major less than 2 GB; configurable under Settings > Parameters: opnsense_update_min_free_gb / opnsense_update_min_free_gb_major, agent 3.17.8 or later, older agents check 15 %)step fails, no retry
opnsense-update/pkg already running (also orphaned)wait up to 10 min, then retry
Agent dies mid-updateresult interrupted on reconnect, retry (the update is idempotent)
Agent still busy with the previous attempt (busy)retried like transient
Time budget exceeded (150 min, major 225 min - above every agent timeout)step timeout, no second start on the node, run failed
Major: target version is not offeredthe chain stops on the last release reached (target version is not offered), partner untouched
Major: firmware check between two steps failsretried like transient, then firmware check failed
configd not answering (CARP state unknown)step transient, retried - no update without a known maintenance state
Node does not come back after the rebootrun fails, peer stays MASTER, second node is not touched
Partner goes offline or fails before (or between the steps of) the second nodesecond node is not rebooted - skipped or chain stopped with the reason
Command hangs (mirror does not answer)the agent kills the whole process group at the phase timeout; the phase ends (agent 3.48.3, before a hanging pkg could block the update forever)
Node back but maintenance stuckremediation task, health check again
Roles not MASTER/BACKUP (INIT, split brain)pair skipped before anything is changed
Schedule starts > 30 min late (backend/worker was down)run missed, nothing executed - no failover at 9 in the morning
Maintenance window exceededno new steps, run incomplete
Second schedule or manual click on the same firewall or its HA partnerwaits or is rejected with HTTP 409
Agent auto-update or reboot task during the updaterejected/deferred
Backend restart in the middle of a runstate lives in the database, the next tick continues

Reading the Result (Jobs Tab) ​

  • Banner at the top: active runs and runs of the last 24 h with problems.
  • Status dot of the schedule: green = success, orange = partial/incomplete/missed, red = failed.
  • Expand a row: the last five runs with step chips (gw2 BACKUP ✓ → gw1 MASTER ✗ health_timeout).
  • Click a run: details with phases, versions, error class, full output and live progress.
  • Resume: plans only the failed/skipped nodes as a new run; roles are re-read from the CARP state.
  • Abort: no new steps. A task the agent has not fetched yet expires immediately (it is never executed); a step already running on the agent is still verified to the end. The run then finishes with status aborted and a summary.

In the Logs tab the individual tasks carry a Run #n badge.

API ​

EndpointPurpose
GET /api/update-runs?schedule_id=&status=&firewall_id=&limit=list runs
GET /api/update-runs/<id>run with steps and task results
POST /api/update-runsstart a run now (firewall_ids / host_group_id / customer_id, update_type, ha_mode, include_peers)
POST /api/update-runs/<id>/abort · /resumeabort / resume
POST /api/firewalls/<id>/ha-peerset the peer manually (peer_id, locked)
POST /api/firewalls/<id>/carp-maintenancemaintenance on/off (`state: on

Major Upgrade across several releases (from 3.17.0) ​

OPNsense always offers only the next major release (Community: series YY.1 in January, YY.7 in July; Business: YY.4 and YY.10). From 23.7 to 26.7 that is six steps. A major upgrade run does all of them in one go:

  1. Regular update with reboot (all pending packages of the current release)
  2. Check (OPN_UPGRADE_CHECK): which major release does the mirror offer now?
  3. Upgrade to that release (OPN_MAJOR_UPGRADE, two reboots), verified as always
  4. back to 1. until the target is reached, or with Latest until no newer release is offered
Target versionEnds when
Latest (default)no newer major release is offered - the firewall is up to date
YY.1 / YY.7 (Community), YY.4 / YY.10 (Business)that release is reached; if it is not offered on the way, the chain stops with a reason

A fixed target applies only to firewalls of the same edition (since 3.32.4). The run skips firewalls of the other edition with the reason target version does not match the edition. For mixed jobs choose Latest: every firewall gets the newest release of its own edition.

The run detail shows the current step per firewall (Step 4: upgrade to 25.1) and the list of completed steps. On HA pairs the nodes take turns per major release: the BACKUP first (e.g. to 24.1), then the MASTER to 24.1, then the BACKUP to 24.7 and so on. Both nodes never work at the same time and the pair is at most one major release apart. If one node fails, the other one stays where it is.

When the maintenance window is over, the chain stops cleanly after the current step (run incomplete, the next run continues). Exception: if one node of a pair is one major release behind, it still catches up - the pair ends on the same major release. Steps follow each other without a gap; the firewall accepts no other update tasks in between.

As a recurring job: a job Major Upgrade → Latest (e.g. monthly) keeps firewalls on the newest release, across new major releases too. If nothing new is out, it only installs pending updates.

Not as a playbook: playbook steps create tasks directly - no HA sequence, no lock and no waiting for the reboot. Run major upgrades through a job or the Major Upgrade button in the Updates tab (it starts a run as well, target Latest preselected).

Notes:

  • Before the step to 26.1 / 26.7 look at the checks OPNsense Legacy firewall rules and OPNsense Legacy IPsec (OPNsense): legacy IPsec then needs the plugin os-strongswan-legacy.
  • The agent checks in its preflight that the mirror offers exactly the next release (upgrade_major_version) and that at least 2 GB are free.
  • Expect 20-45 minutes per step and node (download, offline installation, two reboots); 23.7 → 26.7 on an HA pair takes several hours.

Checking HA synchronisation (from 3.47.0) ​

OPNsense does not copy the configuration to the second node by itself: the master sends the selected sections via XMLRPC to the backup node when you synchronise under System › High Availability › Status (or a cron job does). If that is forgotten, the backup node runs with old rules after a failover. Control now checks this itself – without any task on the firewall, only from the stored config versions (sub-tab Backups).

What is compared: exactly the sections OPNsense synchronises – no more, no less. Control reads in the hasync block of the master's config.xml which services are ticked for synchronisation (syncitems, on older versions the single synchronize… switches) and compares the matching sections of the newest config versions of both nodes. The service → section mapping comes from the OPNsense source code (src/etc/rc.filter_synchronize and the *_xmlrpc_sync() functions in src/etc/inc/plugins.inc.d), for example:

Service in OPNsenseCompared section
Aliasesaliases, OPNsense.Firewall.Alias
Firewall rulesfilter, OPNsense.Firewall.Filter.rules
NATnat and the MVC NAT rules (outbound, 1:1, NPT)
Virtual IPsCARP VIPs (virtualip, only VIPs with a VHID)
Users and groupssystem.user, system.group
Certificatescert, ca, crl
IPsec, WireGuard, OpenVPN, Unbound, Kea …the respective section

What does not count: hostname, interface addresses and everything else OPNsense does not synchronise; the advskew of CARP VIPs (OPNsense raises it by 100 on the backup itself); entries marked do not sync (nosync); values that exist only on the backup in simple sections OPNsense merely overwrites. Lists (rules, aliases, users, VIPs …) are replaced as a whole by OPNsense – an alias that exists only on the backup is therefore a difference. For legacy rules the order counts.

Who is the master? The node with a sync target (synchronizetoip) and at least one service – only that node sends its configuration. If both nodes are set up like that, the current CARP role decides.

Result per pair (in the HA box on the Updates tab, in the list as a small ≠ on the HA pill):

StatusMeaning
in syncall checked sections are equal
differsat least one section differs – with the list of sections and examples (only paths such as …alias[Admins] abweichend, never values)
unknownno config version on one node, versions older than 26 hours, no synchronisation set up, master not clear, or only plugin sections Control does not know are synchronised

Plugin services (e.g. HAProxy, ACME) are unknown to Control; they are listed as not checked.

When it is checked: as soon as one of the nodes stores a new config version, one hour after the nightly backup and otherwise every 6 hours. Check now in the HA box recomputes immediately from the stored versions. During the first 24 hours of a deviation Control requests a backup from both nodes every 2 hours (trigger HA check; the agent only sends the hash if nothing changed) – so a sync done later is noticed within the hold time. After that only the nightly backup counts.

Compare pair opens the comparison of both nodes (master on the left) and needs the Read config backups permission.

Alert: if a pair differs longer than alert_ha_sync_hold_hours (default 6 hours), Control raises a warning HA sync (see Notifications).

EndpointPurpose
GET /api/firewalls/<id>/ha-syncstate of the pair (status, reason, sections, time)
POST /api/firewalls/<id>/ha-sync/checkrecheck now (admin/ops)

Possible later

The agent does not yet report failures of the XMLRPC sync itself (e.g. wrong password at the sync target). An agent extension could add that (scan the system log for An error occurred while attempting XMLRPC sync); until then such a failure shows up as differs.

DATAZONE Control Documentation