Skip to content

Health Checks ​

Health checks monitor the condition of your hosts through automated verifications. You define checks once and distribute them to any number of hosts via jobs.

Concept ​

A health check consists of:

  1. Check Definition - What is being verified? (Name, category, scripts)
  2. Severity Mapping - How are exit codes interpreted?
  3. Auto-Remediation - Optional automatic remediation
  4. Check jobs - Which hosts are checked and when?

Managing Checks ​

Access check management via the Checks navigation item.

Creating a Check ​

  1. Click Create Check
  2. Configure the check:
FieldDescriptionRequired
NameName of the checkYes
DescriptionWhat does this check verify?No
CategoryOrganizational category (freely selectable)No
ModuleWhich modules the check is available forYes
ScriptsOne or more scripts from the libraryYes
EnabledWhether the check can be executedYes

Categories ​

Categories are used to organize your checks. The category name is freely selectable, common examples:

CategoryTypical Checks
diskDisk usage, SMART status
memoryRAM utilization, swap usage
servicesService availability, process checks
securitySecurity policies, certificate expiration
customUser-defined checks (default)

Severity Mapping ​

The severity mapping assigns a severity level to each exit code:

Exit CodeSeverityMeaning
0OKCheck passed
1WarningWarning
2CriticalCritical
3+UnknownUnknown

Nagios-Compatible

The mapping follows the Nagios plugin convention. Existing monitoring scripts can be reused directly.

Predefined Checks ​

DATAZONE Control creates predefined checks for OPNsense: IPsec MSS-Clamping (see OPNsense) as well as OPNsense Legacy-IPsec and OPNsense Legacy-Firewall-Regeln (see Legacy Migration). They are created once and never overwritten, so your own changes are kept. Auto-remediation is off.

For macOS (since v3.22.0) there are predefined checks for FileVault off and SIP off (critical) as well as firewall off, Gatekeeper off, security update pending for more than 14 days and battery service recommended (warning), see macOS.

Multi-Script Checks ​

A check can contain multiple scripts that are executed sequentially:

  • Scripts are executed in the defined order
  • When Stop on Failure is enabled (default: Yes), execution is aborted on the first error
  • The overall status is determined by the worst individual result

Standard checks without a script (since v3.31.0) ​

Common checks need no script. In the check dialog choose a standard check under Check type and enter thresholds; Control generates the right script for Windows (PowerShell), Linux/PVE/PBS, OPNsense (FreeBSD) and macOS itself. No agent update is needed.

TypeParametersResult
CPU usagewarning/critical from %, measuring timeaverage over the measuring time, field CPU_PERCENT
Memorywarning/critical from %used memory, field MEMORY_PERCENT
Disk spacewarning/critical from %, drive/mount point (empty = all)the fullest drive counts
Pingtarget, count, loss thresholds, optional latencyfields PING_LOSS, PING_MS
Service runningservice name, automatic restartstopped = critical; restarted = warning
Process runningprocess name, minimum counttoo few = critical

Standard checks are distributed via jobs like all checks. The values end up in the result and can be mapped to custom fields.

Predefined standard checks (since v3.31.2) ​

For every module (Windows, Linux, PVE, PBS, OPNsense, macOS) Control creates three ready-made standard checks. You find them under Automation → Checks and in the job form:

CheckWarningCriticalAlert after
CPU usage (10 s average)90 %95 %3 failures in a row
Memory90 %95 %2 failures in a row
Disk space (all drives)85 %95 %1 failure

They only run once a job distributes them (Jobs → New job → action Check). Only then do they show up on the host under Checks. You can adjust the thresholds in the check. Deleted standard checks are not created again. If a module already had its own check of the same type, Control does not add another one.

Windows events and disk health (since v3.69.0) ​

Windows events (Windows only) counts events in the event log (Get-WinEvent) and shows the last five:

ParameterMeaning
LogSystem, Application, Security or e.g. Microsoft-Windows-Windows Defender/Operational
Event IDscomma separated, empty = all
Sourcesoptional, e.g. disk, Ntfs
LevelsCritical, Error, Warning
Period5 minutes to 7 days, match the job (hourly job = 60 minutes)
Warning from / Critical fromnumber of events (critical 0 = never)

Examples: disk errors (System, IDs 7, 51, 153), unexpected reboot (System, ID 41), locked out account (Security, ID 4740), Defender detection (Defender/Operational, IDs 1116, 1117). Field EVENT_COUNT.

Disk health (SMART) checks the physical disks: Windows via Get-PhysicalDisk, reliability counters and SMART prediction, Linux/Proxmox/PBS/OPNsense via smartctl (package smartmontools; if missing the check reports unknown with a hint), macOS via the diskutil SMART status. Critical for a failed disk or predicted failure, warning for wear (default from 80 %), temperature (from 60 degrees), reallocated or pending sectors and read errors. Virtual disks without SMART count as ok. Field DISK_HEALTH. The check is predefined for Windows, Linux, Proxmox, PBS and macOS; like every check it runs through a job.

Alert only after several failures (since v3.31.0) ​

Alert only after … failures in a row (default 1 = immediately) applies to notifications and automatic remediation. With 3, an alert is only sent when the last three runs of the check on the computer were not OK – short spikes no longer cause alerts. Runs of remediation or all-clear scripts do not count.

Script on all-clear (since v3.31.0) ​

Optionally a script runs once when the check is OK again on a computer after a failure – e.g. to end a maintenance mode or close a ticket. The counterpart is automatic remediation (script on failure).

Auto-Remediation ​

Automatic remediation executes a remediation script when a check fails.

Configuration ​

FieldDescriptionDefault
Remediation enabledEnable/disable auto-remediationNo
Remediation scriptScript to execute on failure-
Max. attemptsMaximum number of remediation attempts per failure series (1-10)3
CooldownMinimum time between remediation attempts (in seconds, min. 60)300 (5 min.)

Workflow ​

  1. Check is executed and fails (Status: Warning or Critical)
  2. The consecutive failures counter is incremented
  3. When the failure count reaches the Grace Failures limit (default: 1):
    • And the maximum attempt count has not been exceeded
    • And the cooldown since the last attempt has elapsed → The remediation script is executed
  4. The remediation script runs with elevated priority
  5. On the next successful check, all counters are reset

Example ​

Check: "Nginx is running"
Script: Checks if nginx is active (exit 0 = ok, exit 2 = critical)
Remediation script: systemctl restart nginx
Max. attempts: 3
Cooldown: 300 seconds

Workflow:
1. Check returns "Critical" → Counter: 1/1 (Grace reached)
2. → Remediation: nginx is restarted (attempt 1/3)
3. Next check: Still "Critical" → Counter: 2
4. → Check cooldown: 300s elapsed? → Remediation (attempt 2/3)
5. Next check: "OK" → All counters reset

Distributing Checks (Check Jobs, since v3.8.0) ​

Checks are distributed via jobs, just like updates and scripts. One check job covers any number of hosts, e.g. all firewalls with a single entry.

  1. Open the module (e.g. OPNsense) and its Jobs tab
  2. Click New job and select the action Check
  3. Configure:
FieldDescription
CheckWhich check to run
TargetAll hosts of the module (including hosts added later), a customer, a group or a single host
ScheduleWhen the check runs

Schedule Options ​

TypeDescription
OnceSingle execution at the scheduled time
DailyEvery day at the specified time
WeeklyOn the selected weekday at the specified time
MonthlyOn the selected day of the month
CronCustom cron expression (e.g. */5 * * * * for every 5 minutes)

Summary in the Jobs List ​

For check jobs the jobs list shows the latest result across all target hosts, e.g. 3 OK, 70 not OK, 4 no result. Show affected switches to the host list and filters it to the hosts where the check is not OK.

Migration from assignments

Up to v3.7 checks were assigned per host. With v3.8.0 all existing assignments were converted into check jobs automatically (name Check: <check> (Zuweisung #<no>)). Schedule, target and active state are kept. New assignments can no longer be created. Exception: group assignments for OPNsense, Linux and Windows were only deactivated, because the scheduler never executed them. Create a check job with target group if needed.

Checks in the Host List ​

Checks Column ​

The host lists of all modules have a Checks column. It shows one colored chip per check with the latest result (green OK, yellow warning, red critical, grey unknown). The tooltip shows the time and the result's FIELD: values, e.g. SUGGESTED_MSS.

Since v3.11.0 the column updates live: a new result appears in the list and in the Checks tab of the host panel without reloading.

Searching by Check Status ​

The host list search understands the column directly. The status always refers to the same check:

SearchFinds hosts where ...
checks:mssa check with "mss" in its name has a result
checks:mss=okthat check was OK last time
checks:mss!=okthat check was not OK last time (warning, critical, unknown)
checks:mss=warning,criticalthat check was warning or critical
checks:=criticalany check was critical
"checks:IPsec MSS-Clamping!=ok"check names with spaces: quote the whole term

check: works as well as checks:. Hosts without a result for the check are not matched by !=.

Free text is not enough

Searching for mss critical also finds hosts where MSS is OK and another check is critical. Always use checks:name=status for status queries.

Working on Affected Hosts ​

  1. In the jobs list click Show affected on the check job (or search e.g. checks:mss!=ok)
  2. Enable Bulk and select all shown hosts with the header checkbox
  3. Run the matching script via Script
  4. Done: if the script is the check's remediation, the check runs again automatically (see Re-check). For other scripts run the check again via Check

Visible hosts only

Since v3.8.0 bulk actions only apply to selected hosts that are currently visible in the list. Hosts selected earlier and hidden by a new filter are no longer included.

Re-check After Scripts (since v3.11.0) ​

After a successful script Control checks automatically whether it worked. The re-check runs when all of these apply:

  • The script is the remediation script of an active check, e.g. IPsec MSS-Clamping aktivieren for the check IPsec MSS-Clamping
  • The latest result of that check on the host was not OK
  • The script ended with exit code 0

This applies to scripts from the bulk action, the scripts page, jobs and auto-remediation. Update-run steps and playbooks do not trigger a re-check. A re-check never triggers another one. The result shows up live in the Checks column.

Passing Check Values to Scripts (since v3.11.0) ​

A script can take a value from the host's latest check result. Put this line in the script:

bash
# DZ-CHECK-VAR: MSS=SUGGESTED_MSS
MSS="${MSS:-1350}"

When the agent picks up the task, Control sets MSS right below the shebang to the value of FIELD:SUGGESTED_MSS from the host's newest check result that contains this field. Each host gets its own value. Without a matching result nothing is inserted and the script's default applies.

  • Shell and Python scripts are supported. PowerShell is left unchanged because param() must be the first statement
  • Only plain values made of letters, digits and . _ : / @ + - are used. Other values are dropped
  • The script also gets DZ_CHECK_VAR_<NAME>, e.g. DZ_CHECK_VAR_MSS=SUGGESTED_MSS. It tells the script that the value came from a check
  • The inserted value is stored in the task parameters under check_vars, with the check result and time

Custom Field Mappings ​

Checks can write script outputs to custom fields. To do this, the check script must output special lines:

Script Output Format ​

bash
#!/bin/bash
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
echo "FIELD:disk_usage=$USAGE"

if [ "$USAGE" -gt 90 ]; then
  echo "CRITICAL: Disk usage at $USAGE%"
  exit 2
else
  echo "OK: Disk usage at $USAGE%"
  exit 0
fi

Lines with the prefix FIELD: are parsed by the backend:

  • Format: FIELD:fieldname=value
  • The fieldname is mapped to a custom field via the check's Custom Field Mappings
  • The value is automatically saved in the corresponding custom field of the host

Results ​

In the Host Detail ​

In the Checks tab of the detail panel, you can see:

  • The latest result per check with severity color
  • The check jobs covering this host (or "Run manually")
  • Timestamp of the last check
  • A Run now button per check
  • The latest results with output and triggered remediation

Color Coding ​

ColorSeverityMeaning
GreenOKEverything is fine
YellowWarningAttention required
RedCriticalImmediate action needed
GrayUnknownCould not be verified

Result Details ​

Each check result contains:

  • Status (ok/warning/critical/unknown)
  • Exit code of the script
  • Stdout - Standard output
  • Stderr - Error output
  • Custom Field Updates - Which fields were updated
  • Remediation triggered - Whether auto-remediation was triggered
  • Start/End time and duration

Retention

Check results are automatically deleted after 30 days.

Execution ​

Checks are executed:

  • Scheduled according to the schedule of the check job
  • Manually via "Run now" in a host's Checks tab or via the bulk action Check in the host list
  • After updates as a health check in update schedules

Online hosts only (since v3.57.9)

Recurring check jobs only check hosts that are online when the job runs. Switched-off hosts no longer get a waiting task; they are included again in the first run after they come back. One-time check jobs still wait for offline hosts. "Run now" still creates the task for offline hosts too.

Example Checks ​

Check Disk Usage ​

bash
#!/bin/bash
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$USAGE" -gt 90 ]; then
  echo "CRITICAL: Disk usage at $USAGE%"
  exit 2
elif [ "$USAGE" -gt 80 ]; then
  echo "WARNING: Disk usage at $USAGE%"
  exit 1
else
  echo "OK: Disk usage at $USAGE%"
  exit 0
fi

Check Service ​

bash
#!/bin/bash
SERVICE="nginx"
if systemctl is-active --quiet $SERVICE; then
  echo "OK: $SERVICE is running"
  exit 0
else
  echo "CRITICAL: $SERVICE is not active"
  exit 2
fi

Check Certificate ​

bash
#!/bin/bash
DOMAIN="example.com"
DAYS=$(echo | openssl s_client -connect $DOMAIN:443 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
REMAINING=$(( ($(date -d "$DAYS" +%s) - $(date +%s)) / 86400 ))

if [ "$REMAINING" -lt 7 ]; then
  echo "CRITICAL: Certificate expires in $REMAINING days"
  exit 2
elif [ "$REMAINING" -lt 30 ]; then
  echo "WARNING: Certificate expires in $REMAINING days"
  exit 1
else
  echo "OK: Certificate valid for $REMAINING days"
  exit 0
fi

Behaviour since v3.14.2 ​

  • Severity mapping also applies to exit codes ≠ 0. The agent now always reports the exit code; exit 1 becomes warning, exit 2 critical. Before, every check with exit ≠ 0 was critical. Critical is only forced without an exit code (timeout, start failure).
  • Stuck check tasks leave a result unknown instead of keeping the last (green) state.
  • Re-check also exists for checks with module all.
  • Run check now and check results are bound to module permission and customer scope.
  • Timeouts of system scripts are applied on updates (the IPsec MSS check now has 900 s).

Required software (since v3.49.0, Windows) ​

Built-in check type Required software: list programs by display name (as in "Apps & features", a part is enough) and optionally the Chocolatey or winget package. If a program is missing, the check is critical. With Install missing Control installs it right away (setting up Chocolatey if needed) and checks again. As a check job on a group this keeps the standard software of all workstations up to date.

Names and package IDs are limited to letters, digits and a few characters (no quotes, placeholders or options).

DATAZONE Control Documentation