Monitoring and troubleshooting

Operator-facing guide for Pixop Prime on AWS Marketplace (guest: Amazon Linux 2023) and KVM (same guest; hypervisor: Ubuntu with sudo ./scripts/pixop-install from the extracted installer archive root). This page is the navigation hub: scope, a five-minute diagnostic flow, how to read “healthy” without false positives, and where to go next.

Documentation baseline: behavior described here reflects relay-capable and native appliance paths where applicable. CLI subcommands and timers vary by AMI recipe; confirm with pixop-version, command -v <tool>, and -h on the appliance. When behavior differs, treat the running binaries under /opt/pixop/bin as authoritative.

Scope

Facts vs recommendations: tables labeled Operational fact come from shipped scripts, systemd units, or defaults in /etc/pixop-live. Recommendation rows are Pixop operator practice unless cited to a specific command flag.

Operational success (no invented SLAs)

Success is context-dependent — a standby encoder with no scheduled event is healthy when idle.

Decision hierarchy (do not skip steps): systemd active ≠ UDP listener bound ≠ SRT peer connected ≠ MPEG-TS on the internal on-guest multicast TS backbone ≠ GPU enhancement ≠ correct program at the receiver. Each layer has distinct tools; jumping to pixopctl restart before locating the failing layer wastes airtime and can disturb ingress.

Permissions

Packet capture (pixop-flow) and support bundle run as root on the guest by design.

Five-minute diagnostic flow

Run on the guest (SSH). KVM: run host pixop-install checks in parallel if the VM misbehaves before boot.

pixop-version
pixop-preflight
pixop-stack status
pixopctl status
tsduckctl check main
tsduckctl check egress-main
pixopctl health
pixop-flow --secs 5

Optional AWS entitlement pass:

pixop-license doctor

Read-only preference: use pixopctl health, curl to localhost metrics, and journalctl before pixopctl recover, pixop-stack restart, or manual pixop-healthcheck (the oneshot can trigger recovery when thresholds are met — pixop-healthcheck).

Pipeline checkpoints

[ SRT ingest :5000 ] → [ Ingress -> Pixop :5500 ] → Pixop LIVE → [ Pixop -> replacement :5501 ] → [ Final merged TS :5600 ] → [ SRT main egress :5001 ]
         input                              processing                                    egress

TSDuck does not archive MPEG-TS in journald. For elementary-stream inspection use tap helpers — Stream diagnostics.

Startup, warmup, and idle traffic

Operational facts:

• First AWS boot: pixop-instance-init.service enables units and writes /var/lib/pixop-live/instance-init.complete; until then passphrases and listeners may be incomplete — preflight warns instead of failing some UDP checks.
• pixop-healthcheck.timer is enabled during appliance provisioning and again at end of instance-init (OnBootSec=30s, OnUnitActiveSec=15s, AccuracySec=5s, Persistent=false).
• After pixop-live becomes active, healthcheck ignores failures for GRACE_SECS=120 (from /etc/pixop-live/healthcheck.env).
• Idle / waiting-for-input (awaiting_job, or zero frames with zero libav errors) is not a restart condition for automatic recovery.

Recommendations:

• After cold start, keep a continuous source while GPU models load before judging enhancement.
• New EBS volumes may hydrate slowly on first launch; default launch templates can request provisioned initialization (see AWS Marketplace quickstart). Use pixop-startup-timeline for milestone timing — pixop-startup-timeline.
• Zero Mbps on all pixop-flow stages with no sender is normal; verify with an active encoder before opening a transport ticket.

Automatic recovery (summary)

Operational fact: when PIXOP_HEALTHCHECK_RESTART=1, each timer tick classifies one bad check as either (a) non-idle status=ERROR from metrics JSON (or Prometheus equivalent), or (b) both /v1/metrics.json and /v1/metrics empty for longer than NO_METRICS_RESTART_AFTER_SECS=180 since the last good metrics fetch. BAD_CONSECUTIVE=3 such ticks (after GRACE_SECS=120 post–pixop-live active) may invoke pixopctl recover: stop primary ingress (main), restart pixop-live, start ingress and wait for UDP 5000. Egress stays up. RESTART_COOLDOWN_SECS=120 gates repeat recoveries. Idle / awaiting_job suppresses restart even when status=ERROR; DEGRADED resets the bad counter and does not recover. Timer fires about every 15 s (AccuracySec=5 coalescing, boot delay) — do not treat interval × 3 as a fixed SLA.

Limitation: the missing-metrics branch timestamps from last_ok; if that file is absent the script defaults last_ok to “now”, so metrics that never succeeded since boot do not alone accumulate the 180 s missing timer — use external monitoring for guaranteed endpoint availability. Details: Monitoring and health signals.

Safe recovery escalation

Default stance: observe, then recover the smallest scope that matches evidence. Restarts drop SRT sessions on affected legs.

Do not use docker restart / docker kill on pixop-live. Image changes: pixop-image-update, not pixopctl pull.

AWS first boot / license: if pixop-license doctor reports account-level License Manager setup, fix IAM/account prerequisites with Pixop support guidance. Recommendation: do not restart pixop-instance-init.service on production AWS instances until support confirms — init is mutating and can disturb entitlements.

Triage matrix

Step-by-step fault isolation: Incident response runbook.

Logs and metrics (quick map)

Full signal taxonomy: Monitoring and health signals. In brief: live.log and pixopctl logs for Pixop; tsduckctl logs for TSDuck; localhost 8080 serves /v1/metrics (Prometheus), /v1/metrics.json, and /v1/control/status. Typical on-guest curl to 127.0.0.1 works without credentials; when NVEnc REST auth is configured (restapi.yaml), the same endpoints may require Basic or Bearer — restrict 8080 to management CIDRs and supply credentials to external monitors. Never expose 8080 to untrusted networks.

Support bundle and privacy

pixop-support-bundle

Writes /var/lib/pixop-live/support/pixop-support-<host>-<UTC>.tar.gz (mode 0600). The script excludes license JWT bodies and Pulse env from packs and runs pixop-redact over staged content — best-effort, not a guarantee that no secret ever appears. Recommendation: manually review archives before upload; never attach pixop-license token or tsduckctl credentials output.

Include for support: pixop-version, incident timestamps (UTC), deployment path (Region/instance type or KVM), symptom layer (input / processing / egress), and the bundle path.

Service order reference

Operational fact — two different stop paths:

tsduck-in@main is not PartOf pixop-live — a routine pixopctl restart leaves primary SRT ingest running.

Related documentation

Monitoring and health signals · Incident response runbook · Network and transport · Transport troubleshooting · CLI reference · Connect your first SRT stream