Incident response runbook

Practical fault isolation for Pixop Prime operators. Each section follows symptom → evidence → common causes → safe immediate step → escalation. Perform steps in order; prefer observation before restart.

Assumptions: SSH to the guest as ec2-user. KVM host issues use sudo ./scripts/pixop-install from the extracted installer archive root on Ubuntu — KVM troubleshooting guide. Transport and firewall work: Transport troubleshooting and Network and transport.

Version gate: CLI subcommands and relay helpers vary by AMI recipe—verify with pixop-version, tsduckctl --help, and command -v tsduck-transport-info before relay-specific steps.

Record UTC timestamps at symptom start, after each recovery, and when attaching logs.

0. Universal prep (non-disruptive)

pixop-version
date -u
pixop-preflight
pixop-stack status
tsduckctl check main
tsduckctl check egress-main
pixop-flow --secs 5

Note /var/lib/pixop-live/instance-init.complete — if missing, you may still be in AWS first-boot or a workbench image.

pixop-flow stage names (match printed output; ports from /etc/pixop-live/network.env defaults):

Printed label	Default multicast / port
`SRT ingest :5000`	WAN SRT ingest
`Ingress -> Pixop :5500`	`239.0.0.1:5500` internal TS handoff
`Pixop -> replacement :5501`	`239.0.0.1:5501`
`Final merged TS :5600`	`239.0.0.1:5600`
`SRT main egress :5001`	WAN SRT egress

Capture before any restart:

journalctl -u pixop-healthcheck.service -b --no-pager | tail -n 50
tail -n 80 /var/log/pixop-live/live.log

1. No SRT handshake / peer never connects


Symptom	Encoder or decoder never reaches connected state
Evidence	`tsduckctl check` may still pass (listener bound); `pixop-flow` `SRT ingest` = 0; peer logs show timeout or rejection
Common causes	Security group/NACL/firewall; wrong public IP; passphrase/PBKEYLEN mismatch; caller/listener mode mismatch; init not finished (no passphrase)
Safe step	Off-appliance: verify UDP path and credentials via your encoder sheet. On guest: `tsduckctl check main` / `egress-main`; `ss -ulnp \| grep -E ':5000
Escalation	Bundle + `pixop-version`; describe peer product and whether port 5000/5001 probes succeed from outside

Passphrases: use tsduckctl credentials locally only — never paste into tickets.

2. SRT connected but no MPEG-TS (`SRT ingest` zero, `Ingress -> Pixop` zero)


Symptom	SRT session up; no throughput at ingest
Evidence	`pixop-flow` shows 0 on `SRT ingest` and `Ingress -> Pixop`; ingest log quiet or “waiting”
Common causes	Encoder not pushing TS; wrong SRT streamid/socket; bonded/multipath misconfiguration; relay backend not merging
Safe step	Confirm encoder actually transmits MPEG-TS (not raw ES). If supported: `tsduckctl transport main`; else review `/etc/pixop-live/tsduck/main.env` transport settings (no secrets in tickets). Relay ingress: watch `journalctl -u 'tsduck-in@main'` for reset/backpressure — Transport troubleshooting
Outage scope	`tsduckctl restart main` drops ingest SRT — use only after peer is ready to reconnect

3. TS on `Ingress -> Pixop` but no enhancement (`Pixop -> replacement` zero)


Symptom	Internal tap shows Mbps; Pixop replacement port flat
Evidence	`pixop-flow` `Ingress -> Pixop` > 0, `Pixop -> replacement` ≈ 0; `/v1/metrics.json` `input.framesTotal` flat or `output.framesTotal` flat; `live.log` errors
Common causes	Cold GPU/model warmup; license/operational standby; wrong video PID; libav decode errors; `pixop-live` not active
Safe step	`pixopctl status`; keep continuous source ≥1–2 min. Read `live.log` for NVEnc/CUDA/model lines. `curl` control status for `awaiting_job`. Check `realtime.libavErrors.total` in JSON
Outage scope	`pixopctl restart` — Pixop only, SRT ingest stays up. `pixopctl recover` — stops primary ingest then restarts Pixop

Wrong video PID / stale PID mapping


Evidence	Tap traffic OK; decode errors rising; after source change
Safe step	`tsduckctl detect-video-pid main --secs 10` (read-only) — review reported PID
Mutating	`tsduckctl detect-video-pid main --apply` updates `ORIGINAL_VIDEO_PID` and `INPUT_PCR_PID` and restarts ingress — schedule with downstream

GPU init / hydration / slow cold start


Evidence	Long gap in `pixop-startup-timeline`; EBS-first-launch delay; REST server late in `live.log`
Safe step	`pixop-startup-timeline`; wait through hydration (AWS quickstart). Avoid interpreting healthcheck recover during grace (`GRACE_SECS=120`) as failure
Escalation	Bundle + timeline output; instance type and volume initialization settings

4. Enhancement running but egress empty or wrong

4a. `Final merged TS` / `SRT main egress` zero; `Pixop -> replacement` OK


Evidence	`pixop-flow` `Pixop -> replacement` > 0, `Final merged TS` or `SRT main egress` ≈ 0
Common causes	`tsduck-out@egress-main` stalled; merge plugin blocked; egress not started
Safe step	`tsduckctl check egress-main`; `tsduckctl logs egress-main`; `pixop-stack status`
Outage scope	`tsduckctl restart egress-main` — egress SRT drop only

4b. Egress bitrate OK but receiver shows source quality or wrong resolution


Evidence	`SRT main egress` > 0 on appliance; receiver still shows source quality, wrong program, or lower resolution than expected
Common causes	Decoder locked to pre-enhancement program/PID; capture started during passthrough/warmup window; wrong program selection — output codec/profile follows your configured recipe (engineering KVM builds often use HEVC Main 10; public AMI defaults may differ)
Safe step	Inspect the full recording timeline (not only the first seconds). Select the enhanced program/PID after warmup. Early egress may be pre-enhancement TS by design (`tsduck-out-run` packet skip ~2000 ms)
Recommendation	Reconnect downstream after metrics show stable `output.framesTotal`; confirm expected codec in `:8080` UI or `nvencc.args` backup

5. HDR / color wrong


Symptom	Washed out, wrong gamma, or unexpected BT.709 vs BT.2020
Evidence	`live.log` color metadata; `nvencc.args` and UI params; receiver HDR mode
Common causes	Source HLG/PQ vs output signaling; params.json HDR toggles; downstream tone-mapping
Safe step	Document input format from encoder; compare against a privately held `params.json` backup (under `/var/lib/pixop-live/parameters/` — not in pack-config or support bundle). Use pixop-pack-config only for a sanitized `/etc/pixop-live` snapshot. Recommendation: change picture settings through `:8080` UI during maintenance (Picture processing); runtime UI tweaks (class A) do not require `pixopctl restart` — restart only after `nvencc.args` edits (class C)
Avoid	Inventing API calls not in your build’s HTTP docs

6. Low throughput / stutter / `pixop-flow` exit 3


Symptom	Drops, macroblocking, or `pixop-flow` warns backbone capacity
Evidence	Exit 3; `Ingress -> Pixop` at 100% of `TS_TARGET_BITRATE`; utilization % in `pixop-flow`
Common causes	Source bitrate exceeds configured internal backbone; internal on-guest multicast TS pressure; CPU/GPU saturation
Safe step	Lower encoder bitrate or raise `TS_TARGET_BITRATE` in `network.env` only with engineering review — mis-tuning affects merge/CBR. Check `nvidia-smi`
Backbone ceiling	All stages share the guest UDP multicast TS handoff — extreme bitrates may stress the internal path before WAN

7. SRT latency / timing drift


Symptom	Growing delay, AV sync slip
Evidence	Peer-reported latency; timing probe JSON
Safe step	`tsduckctl tune-latency main` (read-only) — compares `PIPELINE_LATENCY` to `pixop-timing-probe --json`
Mutating	`tsduckctl tune-latency main --apply` writes recommended latency (200–1500 ms) and restarts primary ingress — review read-only report first; do not auto-apply in scripts without operator sign-off

8. Healthcheck recover loops


Symptom	Repeated `recovering pixop-live via pixopctl recover` in journal
Evidence	`journalctl -u pixop-healthcheck.service`; `/var/lib/pixop-live/health/bad_count`; metrics JSON showing non-idle ERROR
Common causes	Real decode failure (non-idle `status=ERROR`); sustained missing JSON and Prometheus metrics per healthcheck rules; misclassified failure (rare — idle should suppress)
Safe step	Distinguish idle vs on-air using `pixop-flow` and `framesTotal`. If metrics missing: `pixopctl status`, port 8080 listen. Recommendation: do not disable timer without cause — fix underlying layer
Cooldown	Recoveries respect `RESTART_COOLDOWN_SECS=120`

9. `pixop-live` down, Docker failed, or disk full

Read-only triage before transport or stack restarts. Host-level detail: Monitoring and health signals.

Symptom	Evidence	Safe step	Outage scope
`pixop-live` not active	`pixop-stack status`; `systemctl status pixop-live.service`	`journalctl -u pixop-live.service -b --no-pager \| tail -n 80`; `tail -n 80 /var/log/pixop-live/live.log`	`pixopctl restart` — Pixop only (ingest may stay up). Avoid `docker restart pixop-live`
Crash loop	`Restart=always` in `systemctl status`	`pixop-preflight`; bundle before repeated restarts	Support if log shows GPU/license/config fault
Docker dead	preflight FAIL; no containers	`systemctl status docker.service`; `journalctl -u docker.service -b --no-pager \| tail -n 40`	Entire pipeline until Docker repaired, then `pixop-stack start`
Disk full	`df -h` shows 100% on `/` or `/var`	`du -sh /var/log/pixop-live /var/lib/pixop-live`	May block logging and clean shutdown — free space per policy; do not delete active logs without guidance

10. AWS first boot, license, Marketplace

Symptom	Evidence	Safe step
No passphrases	No `instance-init.complete`; preflight WARN on `SRT_PASSPHRASE`	Wait for init; `systemctl status pixop-instance-init.service` (read-only)
License Manager errors	`pixop-license doctor`	Fix account SLR/IAM with Pixop guidance — avoid unapproved init restarts
Stack not enabled	Units disabled	Support-directed recovery — not manual re-init by default
Slow first Pixop start	EBS hydration	`pixop-startup-timeline`; see AWS Marketplace quickstart

11. KVM-specific quick paths

Symptom	Host evidence	Guest evidence
VM won’t start	`sudo ./scripts/pixop-install doctor` (from extract root)	N/A
`VFIO_MAP_DMA` ENOMEM	memlock / RAM	N/A
No GPU in guest	`status` not `vfio-pci`	`pixop-preflight` NVIDIA FAIL
License valid on disk, logs lag	`license status` hash	Up to ~1h scheduler lag without restart

Full tables: KVM troubleshooting guide.

Controlled recovery reference

Command	What stops	Typical use
`tsduckctl restart main`	Primary ingest	Ingest wedged
`tsduckctl restart egress-main`	Primary egress	Egress wedged
`pixopctl restart`	Pixop container	`nvencc.args` / Pixop-only
`pixopctl recover`	Ingest + Pixop cycle	Healthcheck path; wedged pipeline
`pixop-stack restart`	Full ordered stop/start (primary pair)	Maintenance window
`pixop-stack restart --all`	Includes secondary instances if configured	Aux paths enabled

Never docker kill / docker restart the live container. Image updates: pixop-image-update.

Configuration change discipline

Change	Review	Rollback
`network.env` bitrate/multicast	Engineering	Restore file from backup; `pixop-stack restart`
`tsduck/*.env` transport	Transport configuration if published	Restore env; `tsduckctl restart` affected instance
`nvencc.args`	Output/format	Restore file; `pixopctl restart`
`PIPELINE_LATENCY` via tune-latency	Probe output	Restore previous value in `main.env`; restart ingress

Use pixop-pack-config before edits to capture a sanitized diagnostic archive of /etc/pixop-live — not a private rollback bundle. For rollback, keep separate root-only copies of nvencc.args, tsduck env/args, network.env, and params.json per Configuration change control §3.

Escalation package

pixop-support-bundle

Operational fact: bundle excludes JWT/pulse from packs and runs pixop-redact — best-effort. Recommendation: open the archive on a trusted machine; remove remaining secrets; confirm no credentials dumps.

Include in the ticket:

pixop-version full output
UTC timeline
Deployment (Region, instance type, or KVM + host OS)
Failing layer: input / processing / egress
pixop-preflight summary line
Whether pixop-flow showed bytes at each stage while on-air
Bundle filename under /var/lib/pixop-live/support/

Exclude: passphrases, pixop-license token, raw license.jwt.