Monitoring and health signals

Systematic reference for what to observe on a Pixop Prime appliance, how shipped tooling interprets those observations, and what automatic remediation may do. This page separates signals (read-only facts) from actions (mutating recovery). Pair with Monitoring and troubleshooting for the short diagnostic flow and Incident response runbook for step-by-step faults.

Version gate: parameters below match /etc/pixop-live/healthcheck.env and pixop-healthcheck on qualified appliances. Timer and recover behavior can differ by recipe; confirm with pixop-preflight, systemctl status pixop-healthcheck.timer, and pixop-version.

Signal taxonomy

Signals are grouped from the host upward. Use the lowest layer that explains the symptom before opening HTTP or restarting services.

Layer A — Hypervisor (KVM only)

Guest signals below apply to AWS and KVM alike.

Layer B — Host VM (Amazon Linux 2023 guest)

pixop-preflight exit codes (observable today): 0 = no FAIL; 1 = one or more FAIL. Warnings (e.g. missing license.jwt before KVM licensing) are not failures. The script header documents exit 2 for usage errors, but the installed no-argument CLI path currently ends at 0 or 1 only.

Ephemeral write probes: preflight uses setpriv as UID 1001 to prove write access under /var/lib/pixop-live/parameters and removes /.pixop-uid1001-write-probe — it does not leave probe files on success.

Layer C — systemd units

Operational fact: tsduck-in@main is not PartOf / Requires pixop-live — ingest can stay up across Pixop-only restarts.

pixop-stack status aggregates pixopctl status and tsduckctl status-all.

Layer D — UDP / SRT transport

Two different questions:

1. Is the appliance listening (or caller-ready)? — tsduckctl check NAME [--type in|out]
2. Are bytes moving on the wire? — pixop-flow (requires root/tcpdump; samples 1–30 s, default 2 s)

Bind check (operational fact): prefer tsduckctl check NAME (uses ss plus relay/native policy). When installed, tsduckctl transport NAME or the standalone tsduck-transport-info helper adds bind/backend detail. Native-only recipes may lack the tsduck-transport-info helper — tsduckctl transport may still be present but fails without the helper; fall back to check, ss -ulnp, and instance env under /etc/pixop-live/tsduck/ (do not paste secrets from env files into tickets). For health_check=srt-udp, the primary ingest/egress ports (defaults 5000 / 5001 unless env overrides) must be bound when the unit is active. For srt-caller, UDP listener checks are not applicable.

Reachability (peer connected, firewall open, correct passphrase) is not proven by bind checks or pixop-preflight — external path testing belongs in Transport troubleshooting.

Layer E — Internal on-guest multicast TS backbone

Defaults from /etc/pixop-live/network.env (239.0.0.x UDP multicast on the guest — not the kernel loopback interface, not customer WAN):

Operational fact: pixop-flow exit 0 when measurement succeeds and tap rate is below TS_TARGET_BITRATE (default 50000000 bps in shipped network.env). Exit 3 = at or above backbone capacity at the tap measurement point. Exit 10 = capture failure. Idle source → 0.00 Mbps with exit 0.

Rates are UDP payload Mbps; SRT stages include protocol overhead and retransmissions.

Layer F — Pixop processing (HTTP)

Pixop LIVE serves local HTTP on port PIXOP_METRICS_PORT (default 8080).

Security (operational fact): services bind on the appliance network namespace (--network host for the container). Local curl to 127.0.0.1:8080 is typically unauthenticated. When REST API auth is configured (NVEnc restapi.yaml — Basic password and/or Bearer token), the same routes may reject anonymous scrapes; external monitors need management network reachability plus configured credentials. Recommendation: restrict 8080 to management VLAN/VPC; never expose it to untrusted networks.

Operational vs control state: JSON status / operationalState (metrics) describe runtime health; /v1/control/status state (e.g. awaiting_job) describes pipeline control — healthcheck treats awaiting_job and zero-frame idle as non-failure even when metrics look alarming.

Fields used by pixop-healthcheck (verified in script)

These are the only JSON/control fields appliance auto-recovery relies on:

Prometheus fallback (when JSON missing): pixop_system_status gauge (≥2 → ERROR, 1 → DEGRADED), and first pixop_*input*frames_total for frame count.

The JSON document can include additional sections (output, processing, uptimeSeconds, version, etc.) when runtime metrics exist — see the version-appropriate API reference and metrics schema for your appliance build (match pixop-version to Pixop release notes). Do not build alerts on field names you have not verified on your build; scrape once on a known-good feed and diff against this list.

Example read-only commands:

pixopctl health
curl -sS --max-time 3 http://127.0.0.1:8080/v1/metrics.json | python3 -m json.tool | head -n 40
curl -sS --max-time 3 http://127.0.0.1:8080/v1/control/status | python3 -m json.tool

pixopctl health hits /v1/metrics (Prometheus), not JSON — suitable for a quick liveness glance, not full parsing.

Layer G — Application logs

pixop-pack-logs and support bundles include rotated excerpts — pixop-support-bundle.

pixop-flow reference

pixop-flow
pixop-flow --secs 5   # 1..30 inclusive

Re-invokes via sudo from the installed path (do not run sudo pixop-flow as a different path). If /var/lib/pixop-live/instance-init.complete is missing, output notes startup may be incomplete.

pixop-healthcheck timer policy

Scheduling (systemd)

Enablement (operational fact): Shipped appliances enable pixop-healthcheck.timer during provisioning; pixop-instance-init runs systemctl enable --now pixop-healthcheck.timer on first AWS boot. Preflight FAIL if the timer is disabled on a launched appliance.

Thresholds (/etc/pixop-live/healthcheck.env)

Timer interval vs detection: the timer fires about every 15 s (OnUnitActiveSec=15, AccuracySec=5 coalescing, OnBootSec=30), but one bad tick alone does nothing. Elapsed time to recover depends on grace, how many consecutive bad ticks accumulate, cooldown, and startup — not a fixed “45 s” product SLA.

Classification rules (operational fact)

Each timer run evaluates one failed observation when either:

1. Metrics present, not idle: status=ERROR (from JSON, or Prometheus pixop_system_status ≥ 2) while not in idle/awaiting_job gate, or
2. Metrics absent: both JSON and Prometheus fetches empty, and wall time since last good metrics fetch exceeds NO_METRICS_RESTART_AFTER_SECS (180 s default).

These are alternatives — not a requirement for both ERROR and missing metrics on the same tick.

• Skips entirely if pixop-live inactive (exit 0).
• Idle: pipeline_state=awaiting_job, or frames_total=0 with libav_errors_total=0 — even if status=ERROR, no restart (bad counter cleared).
• DEGRADED: logged; no restart; clears consecutive bad count like a good tick.
• Recover action: after bad_count >= BAD_CONSECUTIVE (3), subject to RESTART_COOLDOWN_SECS — pixopctl recover only; healthcheck timer keeps running; no docker restart.

State directory: /var/lib/pixop-live/health/ (last_ok, bad_count, last_restart).

Missing-metrics limitation (not a product guarantee): when both endpoints are empty, last_ok is read from disk; if missing, the script uses current time as last_ok. Metrics that have never responded successfully since boot therefore do not advance the 180 s missing timer by themselves — monitor /v1/metrics.json availability externally if you need that guarantee.

Action warning: running /opt/pixop/bin/pixop-healthcheck manually can recover when thresholds are met — prefer read-only tools for routine checks.

What to monitor over time

Pixop does not ship a single “golden threshold” document for every deployment. Recommendation: baseline your own environment during a known-good rehearsal:

Use your NOC’s standard SRT metrics (RTT, loss, bandwidth) on encoders/decoders — the appliance does not replace peer-side monitoring.

Mutating helpers (signals vs actions)

See tsduckctl, pixop-timing-probe.

Host faults outside the media path

Use read-only checks first; deeper transport steps live in Transport troubleshooting and Incident response runbook.

pixop-live inactive or crash loop

pixop-healthcheck exits 0 when pixop-live is inactive — it does not start the unit.

Docker daemon failed

Log or disk pressure

Secondary TSDuck instances (when configured): pixop-stack status and tsduckctl status-all — recover only the named instance if evidence points there.

Cross-reference table

Related

• Monitoring and troubleshooting
• Incident response runbook
• pixop-healthcheck · pixop-flow · pixop-preflight