Notifications, Backup SLA & Capacity
Since version 3.9.2 DATAZONE Control reports problems on its own: via Microsoft Teams, ntfy, e-mail or webhook. Two new checks come with it: the backup SLA (when was each VM last backed up?) and the capacity forecast (when will a disk be full?).
Settings: sidebar Administration → Settings → Notifications (administrators only).
What is reported
Control checks every minute:
| Event | When | Severity |
|---|---|---|
| Host offline | no heartbeat for the configured time (default 10 minutes) | critical; offline for more than 7 days only a warning and not sent |
| Checks | latest result of a check is critical or warning | same as the check result |
| Backup SLA | a running VM has no backup within its RPO (see below) | warning up to twice the RPO, then critical; no backup = critical |
| Capacity | a disk, PVE storage or PBS datastore is filling up | warning from 90 % or full in ≤ 30 days; critical from 95 % or full in ≤ 7 days |
| Update runs | an OPNsense update run ends with errors, was missed or skipped firewalls | critical on errors, otherwise warning |
| Certificates (from 3.40.0) | an ACME / Let's Encrypt certificate of an OPNsense expires | warning at 21 days left, critical at 7 days left or expired |
| Gateways (from 3.40.0) | an OPNsense gateway has been offline or had too much loss or latency for 10 minutes in a row | offline = critical, loss > 20 % or latency > 300 ms = warning |
| VPN tunnels (from 3.40.0) | a monitored IPsec tunnel or WireGuard site is down (default 15 minutes in a row) | critical |
| HA sync (from 3.47.0) | the backup node of an OPNsense HA pair differs from the master in synchronised sections (default 6 hours in a row) | warning |
Not reported: switched-off Windows workstations (type Workstation) and check results of hosts that are currently offline. Since v3.12.0 also no OPNsense firewall as offline while it reboots inside an update run (until the health check has ended).
Flow
- New: a problem is reported once. No repeats while it persists.
- Escalation: if a warning becomes critical, a new message is sent.
- Resolved: when the problem is gone Control sends an all-clear (can be switched off per channel). Update runs get no all-clear.
- Batching: everything new within the same minute comes in one message (a network outage does not produce 30 single messages).
- Introduction: on first start all existing problems are taken over silently. Only what is new afterwards is sent.
General settings
| Field | Meaning |
|---|---|
| Notifications active | off = problems only appear in Control (status page, bell), nothing is sent |
| Offline alert after (minutes) | how long a host may be without heartbeat before it is reported (default 10) |
| Default backup RPO (hours) | maximum age of the newest backup per VM (default 24) |
| Quiet hours | time window (German time) during which only critical messages are sent immediately; warnings and all-clears follow afterwards |
| Address for links | base address for links in messages, e.g. https://control.datazone.de. Empty = no links |
Channels
All messages go to shared channels. Per channel you set:
- Send from: warning and critical or critical only
- Events: selection of event kinds; no selection = all
- Send all-clear: yes/no
Test sends a test message immediately. Delivery errors are shown in red on the channel (with time) and disappear after the next successful delivery.
Microsoft Teams
- In Teams, in the target channel: ⋯ → Workflows → "Post to a channel when a webhook request is received"
- Create the workflow and copy the URL
- In Control add a channel of type Teams and paste the URL
The URL is a credential: Control afterwards only shows its beginning. To change it enter a new URL; an empty field keeps the stored one.
ntfy
Server (default https://ntfy.sh or your own), topic and optionally an access token. Critical messages use the highest priority, warnings the normal one. Subscribe to the topic in the ntfy app.
E-mail
Separate recipients with commas. Sending uses the e-mail server (SMTP) on the same tab:
| Field | Note |
|---|---|
| SMTP server, port | e.g. mail.example.com, 587 |
| Encryption | STARTTLS (port 587), SSL/TLS (port 465) or none |
| User, password | leave empty if the server accepts without login. The password is never shown again; empty = keep |
| Sender address | e.g. control@datazone.de |
The same SMTP server is also used for script e-mail notifications.
Webhook
Control sends JSON (subject, text, opened, resolved). With a secret the body is signed: header X-DC-Signature: sha256=<HMAC-SHA256 of the raw body>.
Backup SLA
For every running VM or container on PVE hosts Control checks when the latest backup was taken:
- Agent 3.9.2 or later: every hour the agent reads the newest backup per VM from all backup storages of the host (vzdump archives and Proxmox Backup Server). This is the exact source.
- Agent 3.69.11 or later, job log: the agent also reads the logs of the backup jobs of the last 45 days. If a log says Finished Backup for a VM, that backup counts even if the storage does not show it right now, e.g. USB disks that a hook script only mounts during the job, rotating disks or shares that are only reachable at times. Per VM the newer result of storage and log wins; if it comes from the log, the VM shows from job log. This does not prove that the file on a removed disk is still readable, only a restore test does.
- Older agents: Control checks whether the VM is part of an enabled backup job and uses the host's last successful backup run as the time. VMs in no job are shown as not in any backup job.
Rating: backup younger than the RPO = within RPO; up to twice the RPO = warning; older or no backup at all = critical. Stopped VMs are not rated, neither are hosts that are currently offline. Excluded VMs (see below) do not count in the share.
Setting the RPO
Order:
- Group: field Backup RPO (hours) on PVE groups (Administration → Groups). With several groups the smallest value wins.
- Customer: field Backup RPO (hours) in the customer dialog.
- Default: Settings → Notifications (24 h).
Where it shows
- Status page: tile Backup SLA (share of VMs within RPO), card Backup SLA listing all VMs outside the RPO, column Backup in the customer traffic light, entries in Needs attention
- Bell in the top bar
- Notifications (event Backup SLA)
Clicking an entry opens the PVE host directly on the Backups tab.
Exclusions, mute, take over, maintenance windows
Since version 3.10.0 you control what is reported. All entries are collected under sidebar → Automation → Maintenance & exceptions. Administrators and technicians (permission edit or tasks in the module) may create and change them. A reason is required. Every change is written to the audit log.
| Type | Purpose | Effect |
|---|---|---|
| Deliberately not backed up (exclusion) | VM shall not be backed up, e.g. backed up externally with Veeam, test VM | VM is not rated in the backup SLA and not reported, listed under Excluded. Indefinite by default, expiry date optional |
| Mute | problem is known and being worked on | nothing is sent until the period ends (1 hour to 1 month, a date or indefinitely). The alert stays visible in Control. If the problem still exists afterwards, it is reported as usual |
| Take over | “I am on it” | alert stays visible with the name, but no longer counts in the bell. Ends automatically when the problem is resolved |
| Maintenance window | planned work | for a host, group or customer: once (from–to), daily or weekly (weekdays, time, duration, optional end of series). Nothing is sent during the window, afterwards as usual |
How to
- Status page → Needs attention: menu ⋯ on every alert: Take over, Mute …, for backup alerts Deliberately not backed up …. In the dialog choose what it applies to: only this alert, all alerts of this kind on the host, all alerts of the host or of the whole customer.
- Status page → Backup SLA card: tick VMs (or all), then Deliberately not backed up …, Mute … or Take over for all selected at once. The Excluded tab lists all excluded VMs with reason, name and expiry; Remove takes the exclusion back.
- Muted and taken-over alerts are listed at the bottom under Muted or taken over (expandable).
- Maintenance & exceptions: plan and edit maintenance windows; all exclusions, mutes and take-overs as a list with search, filter by type, extend/edit and remove (also several at once). Expired entries stay visible for 30 days, maintenance and out of service for 365 days (show expired) so they can be renewed.
Excluded automatically
- PVE tag: VMs with the tag
no-backup(configurable under Settings → Notifications → PVE tags “do not back up”, several separated by commas) count as deliberately not backed up. Set the tag on the VM in Proxmox. Requires agent 3.10.0 or later. - Excluded in a backup job: if a VM is on the exclude list of an active backup job and no other job backs it up, it is only shown as info (excluded in the backup job), not as a problem.
If a VM is excluded while an alert for it is open, the alert is closed silently (no all-clear).
Maintenance and out of service (from 3.18.0)
A host can be set to maintenance or out of service: with since (also in the past), until or indefinitely, and a reason.
| State | For | Effect |
|---|---|---|
| Maintenance | temporary, e.g. hardware replacement, rebuild, outage | no alerts, the host is listed in the overview under In maintenance or out of service instead of the problems, yellow dot and badge in the host list |
| Out of service | switched off, decommissioned, spare device in storage | like maintenance; in addition, alerts that are already open are closed silently (no all-clear). Grey dot in the list |
While a one-time state applies, scheduled jobs (updates, scripts, checks) skip the host. Update runs list the firewall as skipped with the reason In maintenance (Control) or Out of service (Control); for an HA pair the whole pair is skipped. Anything started by hand (e.g. an update from the host panel, Run now or a run from the Updates tab) still runs.
Recurring maintenance windows (daily, weekly) only silence alerts and show the host as maintenance. Jobs run normally in these windows, because they are often exactly the windows in which updates are supposed to run.
How to:
- Host list: select hosts → action Maintenance / out of service …. The filter has Maintenance and Out of service under Status.
- Host panel: menu ⋯ → Maintenance / out of service …, End maintenance or End out of service, and Maintenance / service history. The history shows all periods of the host, including windows via group or customer, with reason and name.
- Automation → Maintenance & exceptions: all maintenance and out-of-service entries, also for groups and customers.
End sets the end to now. The entry is kept as history for 365 days. If a state with an end expires and the host is still offline, it is reported normally afterwards.
OPNsense: certificates, gateways and VPN (from 3.40.0)
Control evaluates the data the agent already sends with every heartbeat. No agent update is needed.
Certificates
- Only ACME / Let's Encrypt certificates (as detected by the agent) are reported. Self-signed and other certificates never alert, even when they expire.
- Warning at 21 days left, critical at 7 days left or when expired. Both values can be changed under Settings → Notifications → OPNsense.
- Example:
gw1: Zertifikat portal.example.de läuft in 5 Tagen ab(alert texts are German).
Gateways
- Critical when a gateway is offline; warning above 20 % packet loss or above 300 ms latency.
- An alert is only raised when the state has lasted 10 minutes in a row (hold time). Short glitches stay silent. The hold time survives a restart of Control.
- Gateways without their own rating (status
none,unknown,pending, monitoring off) and forced down gateways are ignored. - Thresholds and hold time under Settings → Notifications → OPNsense.
- History: Control stores the daily maximum of loss and latency per gateway (sampled every 5 minutes). The firewall detail panel, tab Overview → Gateways, shows the maximum of the last 30 days and a small trend line each.
VPN tunnels (opt-in)
VPN alerts only exist for tunnels you monitor. Without an entry Control sends no VPN alert. IPsec tunnels and WireGuard site-to-site links can be monitored, never road-warrior peers and never OpenVPN.
Two ways:
- Single tunnel: in the firewall detail panel, tab VPN, or in the VPN overview (open the tunnel) click Monitor. The pill monitored shows the state. Stop monitoring turns it off again; if a rule covers the tunnel, it is explicitly excluded.
- Rules: Settings → Notifications → VPN monitoring → Add rule. A rule applies to all firewalls, a customer, an OPNsense group or one firewall, to IPsec and/or WG sites and optionally only to tunnels whose name or description contains a text (e.g.
branch).
An exclusion on a single tunnel always wins over rules. A monitored tunnel is reported when it has been down for 15 minutes in a row (configurable per rule or tunnel; with several rules the shortest time applies). A WireGuard site counts as down when none of its enabled peers had a handshake within the last 5 minutes. Example: gw1: IPsec-Tunnel Zentrale getrennt seit 20 min.
Administrators and users with edit or task permission in OPNsense may create entries. Customer-bound users only see and change entries of their customer and cannot create rules for all firewalls or for groups.
Common rules
- Introduction: for each of the three kinds, problems that already exist on the first run (for gateways and VPN until the hold time has passed once) are taken over silently.
- No false all-clear: if the firewall is offline or reboots in an update run, open alerts stay open and hold times keep running. If a tunnel is no longer monitored, its alert closes silently.
- Maintenance / out of service, mute, take over work as for every other alert. Maintenance windows can select the events Certificates, Gateways and VPN tunnels individually. Firewalls in maintenance or out of service are not shown as problems on the status page.
- These alerts also show on the status page and in the bell (with the customer filter). A click opens the firewall in the matching tab.
- Channels can select the events Certificates, Gateways and VPN tunnels individually.
OPNsense: HA synchronisation (from 3.47.0)
For every HA pair Control compares the sections OPNsense synchronises from the master to the backup node (details in Checking HA synchronisation).
- Warning HA sync when the pair has been differs for more than 6 hours in a row (
alert_ha_sync_hold_hours, changeable under Settings → Notifications → OPNsense). Example:HA-Paar gw1 / gw2: Konfiguration weicht seit 7 h abwith the affected sections (e.g. aliases, firewall rules) in the detail. - The alert is raised on the master of the pair; a click opens the firewall on the Updates tab (HA box).
- Unknown (no fresh config versions, no synchronisation set up, master not clear) never alerts and does not close an open alert – no false all-clear. Only a result in sync ends the alert.
- Maintenance / out of service, mute, take over work as for every other alert; it is enough if one of the two nodes is in maintenance (muted) or out of service (alert closes silently). Maintenance windows and channels can select the event HA sync individually.
- Introduction: whatever shows up in the first 26 hours plus the hold time after the update is taken over silently (the first nightly backup of both nodes may only exist by then).
Capacity forecast
Once per hour Control stores the daily maximum usage of
- host disks (all modules),
- PVE storages,
- PBS datastores.
A linear trend is calculated from the last 30 days. From 5 days of data there is a forecast "full in ~X days". The Capacity card on the status page lists all storages with a warning or a forecast up to 60 days, with a small trend line.
Active alerts
At the bottom of the Notifications tab you see what is open right now, with status:
- sent: delivered to at least one channel
- silent: already present at introduction or long known (e.g. host offline for more than 7 days), not sent
- waiting: will be sent with the next run (e.g. after quiet hours)
- muted / maintenance: muted or in a maintenance window; sent afterwards if still open
- taken over by …: someone is on it
Closed alerts are deleted after 60 days.
Behaviour since v3.14.2
- No false all-clears. When a host is briefly offline or one part of the evaluation (checks, backup SLA, capacity) fails in a pass, the affected alerts stay open; before, they were closed, reported "resolved" and reopened minutes later (acknowledgements were lost).
- Delivery is retried. If every channel fails, the alert stays open for up to an hour and is sent again on the next pass; same for all-clears.
- Silent baseline only once. The first pass after installation adopts existing problems silently; this is remembered as a setting and not repeated when the alert table becomes empty through retention.
- Metric errors no longer block alerting (separate error handling).
- Customer-wide exclusions/mutes require, for non-administrators, edit or task rights in every module the customer has hosts in.
- Quiet hours with equal start and end mean no quiet hours; webhook addresses no longer appear in a channel's error text; the backup-SLA job fallback honours a backup job's node; the alert text for update runs counts successful nodes correctly.