Health Checks
Health checks monitor the condition of your hosts through automated verifications. You define checks once and distribute them to any number of hosts via jobs.
Concept
A health check consists of:
- Check Definition - What is being verified? (Name, category, scripts)
- Severity Mapping - How are exit codes interpreted?
- Auto-Remediation - Optional automatic remediation
- Check jobs - Which hosts are checked and when?
Managing Checks
Access check management via the Checks navigation item.
Creating a Check
- Click Create Check
- Configure the check:
| Field | Description | Required |
|---|---|---|
| Name | Name of the check | Yes |
| Description | What does this check verify? | No |
| Category | Organizational category (freely selectable) | No |
| Module | Which modules the check is available for | Yes |
| Scripts | One or more scripts from the library | Yes |
| Enabled | Whether the check can be executed | Yes |
Categories
Categories are used to organize your checks. The category name is freely selectable, common examples:
| Category | Typical Checks |
|---|---|
disk | Disk usage, SMART status |
memory | RAM utilization, swap usage |
services | Service availability, process checks |
security | Security policies, certificate expiration |
custom | User-defined checks (default) |
Severity Mapping
The severity mapping assigns a severity level to each exit code:
| Exit Code | Severity | Meaning |
|---|---|---|
| 0 | OK | Check passed |
| 1 | Warning | Warning |
| 2 | Critical | Critical |
| 3+ | Unknown | Unknown |
Nagios-Compatible
The mapping follows the Nagios plugin convention. Existing monitoring scripts can be reused directly.
Predefined Checks
DATAZONE Control creates predefined checks for OPNsense: IPsec MSS-Clamping (see OPNsense) as well as OPNsense Legacy-IPsec and OPNsense Legacy-Firewall-Regeln (see Legacy Migration). They are created once and never overwritten, so your own changes are kept. Auto-remediation is off.
For macOS (since v3.22.0) there are predefined checks for FileVault off and SIP off (critical) as well as firewall off, Gatekeeper off, security update pending for more than 14 days and battery service recommended (warning), see macOS.
Multi-Script Checks
A check can contain multiple scripts that are executed sequentially:
- Scripts are executed in the defined order
- When Stop on Failure is enabled (default: Yes), execution is aborted on the first error
- The overall status is determined by the worst individual result
Standard checks without a script (since v3.31.0)
Common checks need no script. In the check dialog choose a standard check under Check type and enter thresholds; Control generates the right script for Windows (PowerShell), Linux/PVE/PBS, OPNsense (FreeBSD) and macOS itself. No agent update is needed.
| Type | Parameters | Result |
|---|---|---|
| CPU usage | warning/critical from %, measuring time | average over the measuring time, field CPU_PERCENT |
| Memory | warning/critical from % | used memory, field MEMORY_PERCENT |
| Disk space | warning/critical from %, drive/mount point (empty = all) | the fullest drive counts |
| Ping | target, count, loss thresholds, optional latency | fields PING_LOSS, PING_MS |
| Service running | service name, automatic restart | stopped = critical; restarted = warning |
| Process running | process name, minimum count | too few = critical |
Standard checks are distributed via jobs like all checks. The values end up in the result and can be mapped to custom fields.
Predefined standard checks (since v3.31.2)
For every module (Windows, Linux, PVE, PBS, OPNsense, macOS) Control creates three ready-made standard checks. You find them under Automation → Checks and in the job form:
| Check | Warning | Critical | Alert after |
|---|---|---|---|
| CPU usage (10 s average) | 90 % | 95 % | 3 failures in a row |
| Memory | 90 % | 95 % | 2 failures in a row |
| Disk space (all drives) | 85 % | 95 % | 1 failure |
They only run once a job distributes them (Jobs → New job → action Check). Only then do they show up on the host under Checks. You can adjust the thresholds in the check. Deleted standard checks are not created again. If a module already had its own check of the same type, Control does not add another one.
Windows events and disk health (since v3.69.0)
Windows events (Windows only) counts events in the event log (Get-WinEvent) and shows the last five:
| Parameter | Meaning |
|---|---|
| Log | System, Application, Security or e.g. Microsoft-Windows-Windows Defender/Operational |
| Event IDs | comma separated, empty = all |
| Sources | optional, e.g. disk, Ntfs |
| Levels | Critical, Error, Warning |
| Period | 5 minutes to 7 days, match the job (hourly job = 60 minutes) |
| Warning from / Critical from | number of events (critical 0 = never) |
Examples: disk errors (System, IDs 7, 51, 153), unexpected reboot (System, ID 41), locked out account (Security, ID 4740), Defender detection (Defender/Operational, IDs 1116, 1117). Field EVENT_COUNT.
Disk health (SMART) checks the physical disks: Windows via Get-PhysicalDisk, reliability counters and SMART prediction, Linux/Proxmox/PBS/OPNsense via smartctl (package smartmontools; if missing the check reports unknown with a hint), macOS via the diskutil SMART status. Critical for a failed disk or predicted failure, warning for wear (default from 80 %), temperature (from 60 degrees), reallocated or pending sectors and read errors. Virtual disks without SMART count as ok. Field DISK_HEALTH. The check is predefined for Windows, Linux, Proxmox, PBS and macOS; like every check it runs through a job.
Alert only after several failures (since v3.31.0)
Alert only after … failures in a row (default 1 = immediately) applies to notifications and automatic remediation. With 3, an alert is only sent when the last three runs of the check on the computer were not OK – short spikes no longer cause alerts. Runs of remediation or all-clear scripts do not count.
Script on all-clear (since v3.31.0)
Optionally a script runs once when the check is OK again on a computer after a failure – e.g. to end a maintenance mode or close a ticket. The counterpart is automatic remediation (script on failure).
Auto-Remediation
Automatic remediation executes a remediation script when a check fails.
Configuration
| Field | Description | Default |
|---|---|---|
| Remediation enabled | Enable/disable auto-remediation | No |
| Remediation script | Script to execute on failure | - |
| Max. attempts | Maximum number of remediation attempts per failure series (1-10) | 3 |
| Cooldown | Minimum time between remediation attempts (in seconds, min. 60) | 300 (5 min.) |
Workflow
- Check is executed and fails (Status: Warning or Critical)
- The consecutive failures counter is incremented
- When the failure count reaches the Grace Failures limit (default: 1):
- And the maximum attempt count has not been exceeded
- And the cooldown since the last attempt has elapsed → The remediation script is executed
- The remediation script runs with elevated priority
- On the next successful check, all counters are reset
Example
Check: "Nginx is running"
Script: Checks if nginx is active (exit 0 = ok, exit 2 = critical)
Remediation script: systemctl restart nginx
Max. attempts: 3
Cooldown: 300 seconds
Workflow:
1. Check returns "Critical" → Counter: 1/1 (Grace reached)
2. → Remediation: nginx is restarted (attempt 1/3)
3. Next check: Still "Critical" → Counter: 2
4. → Check cooldown: 300s elapsed? → Remediation (attempt 2/3)
5. Next check: "OK" → All counters resetDistributing Checks (Check Jobs, since v3.8.0)
Checks are distributed via jobs, just like updates and scripts. One check job covers any number of hosts, e.g. all firewalls with a single entry.
- Open the module (e.g. OPNsense) and its Jobs tab
- Click New job and select the action Check
- Configure:
| Field | Description |
|---|---|
| Check | Which check to run |
| Target | All hosts of the module (including hosts added later), a customer, a group or a single host |
| Schedule | When the check runs |
Schedule Options
| Type | Description |
|---|---|
| Once | Single execution at the scheduled time |
| Daily | Every day at the specified time |
| Weekly | On the selected weekday at the specified time |
| Monthly | On the selected day of the month |
| Cron | Custom cron expression (e.g. */5 * * * * for every 5 minutes) |
Summary in the Jobs List
For check jobs the jobs list shows the latest result across all target hosts, e.g. 3 OK, 70 not OK, 4 no result. Show affected switches to the host list and filters it to the hosts where the check is not OK.
Migration from assignments
Up to v3.7 checks were assigned per host. With v3.8.0 all existing assignments were converted into check jobs automatically (name Check: <check> (Zuweisung #<no>)). Schedule, target and active state are kept. New assignments can no longer be created. Exception: group assignments for OPNsense, Linux and Windows were only deactivated, because the scheduler never executed them. Create a check job with target group if needed.
Checks in the Host List
Checks Column
The host lists of all modules have a Checks column. It shows one colored chip per check with the latest result (green OK, yellow warning, red critical, grey unknown). The tooltip shows the time and the result's FIELD: values, e.g. SUGGESTED_MSS.
Since v3.11.0 the column updates live: a new result appears in the list and in the Checks tab of the host panel without reloading.
Searching by Check Status
The host list search understands the column directly. The status always refers to the same check:
| Search | Finds hosts where ... |
|---|---|
checks:mss | a check with "mss" in its name has a result |
checks:mss=ok | that check was OK last time |
checks:mss!=ok | that check was not OK last time (warning, critical, unknown) |
checks:mss=warning,critical | that check was warning or critical |
checks:=critical | any check was critical |
"checks:IPsec MSS-Clamping!=ok" | check names with spaces: quote the whole term |
check: works as well as checks:. Hosts without a result for the check are not matched by !=.
Free text is not enough
Searching for mss critical also finds hosts where MSS is OK and another check is critical. Always use checks:name=status for status queries.
Working on Affected Hosts
- In the jobs list click Show affected on the check job (or search e.g.
checks:mss!=ok) - Enable Bulk and select all shown hosts with the header checkbox
- Run the matching script via Script
- Done: if the script is the check's remediation, the check runs again automatically (see Re-check). For other scripts run the check again via Check
Visible hosts only
Since v3.8.0 bulk actions only apply to selected hosts that are currently visible in the list. Hosts selected earlier and hidden by a new filter are no longer included.
Re-check After Scripts (since v3.11.0)
After a successful script Control checks automatically whether it worked. The re-check runs when all of these apply:
- The script is the remediation script of an active check, e.g. IPsec MSS-Clamping aktivieren for the check IPsec MSS-Clamping
- The latest result of that check on the host was not OK
- The script ended with exit code 0
This applies to scripts from the bulk action, the scripts page, jobs and auto-remediation. Update-run steps and playbooks do not trigger a re-check. A re-check never triggers another one. The result shows up live in the Checks column.
Passing Check Values to Scripts (since v3.11.0)
A script can take a value from the host's latest check result. Put this line in the script:
# DZ-CHECK-VAR: MSS=SUGGESTED_MSS
MSS="${MSS:-1350}"When the agent picks up the task, Control sets MSS right below the shebang to the value of FIELD:SUGGESTED_MSS from the host's newest check result that contains this field. Each host gets its own value. Without a matching result nothing is inserted and the script's default applies.
- Shell and Python scripts are supported. PowerShell is left unchanged because
param()must be the first statement - Only plain values made of letters, digits and
. _ : / @ + -are used. Other values are dropped - The script also gets
DZ_CHECK_VAR_<NAME>, e.g.DZ_CHECK_VAR_MSS=SUGGESTED_MSS. It tells the script that the value came from a check - The inserted value is stored in the task parameters under
check_vars, with the check result and time
Custom Field Mappings
Checks can write script outputs to custom fields. To do this, the check script must output special lines:
Script Output Format
#!/bin/bash
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "FIELD:disk_usage=$USAGE"
if [ "$USAGE" -gt 90 ]; then
echo "CRITICAL: Disk usage at $USAGE%"
exit 2
else
echo "OK: Disk usage at $USAGE%"
exit 0
fiLines with the prefix FIELD: are parsed by the backend:
- Format:
FIELD:fieldname=value - The
fieldnameis mapped to a custom field via the check's Custom Field Mappings - The value is automatically saved in the corresponding custom field of the host
Results
In the Host Detail
In the Checks tab of the detail panel, you can see:
- The latest result per check with severity color
- The check jobs covering this host (or "Run manually")
- Timestamp of the last check
- A Run now button per check
- The latest results with output and triggered remediation
Color Coding
| Color | Severity | Meaning |
|---|---|---|
| Green | OK | Everything is fine |
| Yellow | Warning | Attention required |
| Red | Critical | Immediate action needed |
| Gray | Unknown | Could not be verified |
Result Details
Each check result contains:
- Status (ok/warning/critical/unknown)
- Exit code of the script
- Stdout - Standard output
- Stderr - Error output
- Custom Field Updates - Which fields were updated
- Remediation triggered - Whether auto-remediation was triggered
- Start/End time and duration
Retention
Check results are automatically deleted after 30 days.
Execution
Checks are executed:
- Scheduled according to the schedule of the check job
- Manually via "Run now" in a host's Checks tab or via the bulk action Check in the host list
- After updates as a health check in update schedules
Online hosts only (since v3.57.9)
Recurring check jobs only check hosts that are online when the job runs. Switched-off hosts no longer get a waiting task; they are included again in the first run after they come back. One-time check jobs still wait for offline hosts. "Run now" still creates the task for offline hosts too.
Example Checks
Check Disk Usage
#!/bin/bash
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$USAGE" -gt 90 ]; then
echo "CRITICAL: Disk usage at $USAGE%"
exit 2
elif [ "$USAGE" -gt 80 ]; then
echo "WARNING: Disk usage at $USAGE%"
exit 1
else
echo "OK: Disk usage at $USAGE%"
exit 0
fiCheck Service
#!/bin/bash
SERVICE="nginx"
if systemctl is-active --quiet $SERVICE; then
echo "OK: $SERVICE is running"
exit 0
else
echo "CRITICAL: $SERVICE is not active"
exit 2
fiCheck Certificate
#!/bin/bash
DOMAIN="example.com"
DAYS=$(echo | openssl s_client -connect $DOMAIN:443 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
REMAINING=$(( ($(date -d "$DAYS" +%s) - $(date +%s)) / 86400 ))
if [ "$REMAINING" -lt 7 ]; then
echo "CRITICAL: Certificate expires in $REMAINING days"
exit 2
elif [ "$REMAINING" -lt 30 ]; then
echo "WARNING: Certificate expires in $REMAINING days"
exit 1
else
echo "OK: Certificate valid for $REMAINING days"
exit 0
fiBehaviour since v3.14.2
- Severity mapping also applies to exit codes ≠ 0. The agent now always reports the exit code; exit 1 becomes warning, exit 2 critical. Before, every check with exit ≠ 0 was critical. Critical is only forced without an exit code (timeout, start failure).
- Stuck check tasks leave a result
unknowninstead of keeping the last (green) state. - Re-check also exists for checks with module all.
- Run check now and check results are bound to module permission and customer scope.
- Timeouts of system scripts are applied on updates (the IPsec MSS check now has 900 s).
Required software (since v3.49.0, Windows)
Built-in check type Required software: list programs by display name (as in "Apps & features", a part is enough) and optionally the Chocolatey or winget package. If a program is missing, the check is critical. With Install missing Control installs it right away (setting up Chocolatey if needed) and checks again. As a check job on a group this keeps the standard software of all workstations up to date.
Names and package IDs are limited to letters, digits and a few characters (no quotes, placeholders or options).