pixop-healthcheck

Periodic diagnostic for a running pixop-live.service. Reads local HTTP metrics and control state; on sustained proven failure it may invoke pixopctl recover when automatic restart is enabled. This is an internal appliance script — operators normally observe it via journalctl, not as routine maintenance.

Advanced / support use: Running pixop-healthcheck manually on a production host can trigger ingress teardown and recovery when PIXOP_HEALTHCHECK_RESTART=1 and failure thresholds are met. Prefer pixopctl health, pixop-preflight, and metrics URLs for read-only checks.

Availability: Verify with command -v pixop-healthcheck and pixop-version. Installed binary: /opt/pixop/bin/pixop-healthcheck. Periodic scheduling uses systemd unit pixop-healthcheck.timer when enabled on your recipe; thresholds and restart policy in /etc/pixop-live/healthcheck.env.

Purpose and scope

The timer (OnBootSec=30s, OnUnitActiveSec=15s) starts a oneshot service that:

Skips if pixop-live.service is inactive (exit 0, message only).
Honors a startup grace (GRACE_SECS, default 120) after the unit entered active state.
Fetches http://127.0.0.1:${PIXOP_METRICS_PORT:-8080}/v1/metrics.json (preferred) and /v1/control/status.
Falls back to Prometheus /v1/metrics if JSON is unavailable.
Classifies idle / waiting-for-input (no frames and no libav errors, or pipeline awaiting_job) as healthy even when status is ERROR.
Increments consecutive bad count; after BAD_CONSECUTIVE (default 3) bad checks, may call pixopctl recover if PIXOP_HEALTHCHECK_RESTART=1 in /etc/pixop-live/healthcheck.env and restart cooldown (RESTART_COOLDOWN_SECS, default 120) has elapsed.

Does not docker restart / docker kill, does not stop the healthcheck timer, and does not pull images.

Usage

pixop-healthcheck

Installed path: /opt/pixop/bin/pixop-healthcheck. Non-root runs re-exec via sudo unless PIXOP_HEALTHCHECK_SKIP_ROOT=1 (tests).

Enable scheduling via pixop-healthcheck.timer, not the oneshot unit.

Representative log lines

Message	Meaning
`pixop-healthcheck: OK`	Healthy this interval
`inside Ns start grace window`	Too soon after `pixop-live` start to judge
`DEGRADED (...; not a restart condition)`	Degraded metrics; no recovery
`idle/waiting-for-input ... not a restart condition`	No sender yet; intentional
`BAD (...; consecutive=k)`	Failure counted toward threshold
`automatic restart disabled`	`PIXOP_HEALTHCHECK_RESTART` not `1`
`restart cooldown ... active`	Recent recovery; waiting
`recovering pixop-live via pixopctl recover`	Mutating recovery started

Exit codes: Almost all paths exit 0 (including BAD without recovery, cooldown, and disabled restart). Exit code follows pixopctl recover only when recovery runs.

Environment (defaults from script)

Variable	Default	Role
`PIXOP_METRICS_PORT`	`8080`	Local metrics port
`GRACE_SECS`	`120`	Post-start skip window
`NO_METRICS_RESTART_AFTER_SECS`	`180`	Missing metrics tolerance
`BAD_CONSECUTIVE`	`3`	Failures before recover
`RESTART_COOLDOWN_SECS`	`120`	Minimum time between recoveries
`PIXOP_HEALTHCHECK_RESTART`	`0` in script; `1` in healthcheck.env on appliance	Allow recover

State files under /var/lib/pixop-live/health/: last_ok, bad_count, last_restart.

Operator workflows

Read-only health (preferred):

pixopctl health
curl -sS http://127.0.0.1:8080/v1/metrics.json | head

Investigate automatic recoveries:

journalctl -u pixop-healthcheck.service -b --no-pager
journalctl -u pixop-live -b --no-pager | tail -n 100

Confirm timer and policy:

systemctl status pixop-healthcheck.timer
grep PIXOP_HEALTHCHECK_RESTART /etc/pixop-live/healthcheck.env
pixop-preflight | grep -i healthcheck

Troubleshooting

Symptom	Check
Frequent recoveries with valid SRT idle	Expected if misclassified; capture metrics JSON and control state — idle should not restart
Recoveries during EBS cold start	Long NVEncC delay may be hydration; see startup timeline
`metrics missing`	Pixop not listening on 8080, or container not up — `pixopctl status`
Manual recover needed	`pixopctl recover` (same path healthcheck uses) — see pixopctl

Limitations

Localhost-only HTTP; no WAN probe.
DEGRADED never triggers restart by design.
Recovery recycles primary ingress and pixop-live, not full pixop-stack stop/start.

Monitoring and troubleshooting — automatic recovery summary
pixopctl — recover, health
pixop-preflight — timer enablement checks