Transport troubleshooting

Structured fault isolation for SRT, relay/native backends, internal MPEG-TS multicast, and platform networking. Use with Network and transport for port reference and Transport configuration for safe edits.

Capability gate: tsduckctl transport is always in the CLI wrapper; it needs tsduck-transport-info installed to succeed and then reports native or relay effective backend. tsduck-transport-info bind-status and pixop-net require their respective binaries. Relay-only diagnostics (bonded listeners, stall recovery) apply when TRANSPORT_BACKEND=relay. On native-only images (helper absent), use tsduckctl check, native .env keys, and ss -uln for listeners. Confirm with pixop-version and command -v tsduck-transport-info.

First response (READ-ONLY)

Run on the appliance before restarting services:

pixop-version
tsduckctl check main
tsduckctl check egress-main
pixop-flow --secs 5

When command -v tsduck-transport-info succeeds:

tsduckctl transport main
tsduckctl transport egress-main

Collect logs only if checks are inconclusive:

tsduckctl logs main
tsduckctl logs egress-main
journalctl -u tsduck-in@main -n 100 --no-pager
journalctl -u tsduck-out@egress-main -n 100 --no-pager

Avoid unnecessary tsduckctl restart — restarts clear egress startup drain and interrupt live sessions.

Layer model

Diagnose bottom-up:

  [A] UDP socket bind on 5000 / 5001
  [B] SRT handshake / encryption (passphrase, pbkeylen, mode)
  [C] MPEG-TS packets at 239.0.0.1:5500
  [D] Pixop processing → 239.0.0.1:5501
  [E] Mux / PCR / PID → 239.0.0.1:5600
  [F] Egress SRT to downstream

A success at layer A does not imply B; B does not imply valid TS at C.

Symptom reference

Ingest: `check` shows port not bound

Signals	Likely cause	Safe action
`systemctl` inactive for `tsduck-in@main`	Service failed start; relay template error	`journalctl -u tsduck-in@main`; fix `TRANSPORT_BACKEND` / missing `*.ingest.args`
`TRANSPORT_BACKEND=relay` without relay helper/templates	Relay not available on this image	Revert to `native` or migrate to a relay-capable qualified image
Invalid backend string	Typo in `.env`	Set `native` or `relay` only
Bonded template, partial bind	Local listener IP missing, port in use, or start failure	Multipath and bonded SRT; `pixop-net multipath verify`; `journalctl -u tsduck-in@main`

Ingest: bound but encoder cannot connect

Signals	Likely cause	Safe action
Local bind OK; peer timeout	AWS SG, NACL, on-prem firewall, wrong public IP	Open UDP 5000 from encoder CIDR; verify `tsduckctl credentials` address
KVM userspace lab	Peer targets LAN IP instead of `127.0.0.1` forward	Use forwarded host ports from KVM quickstart
Encryption failure	Passphrase or `pbkeylen` mismatch (16 on ingest)	Re-share credentials sheet securely — do not ticket secrets
Caller/listener inversion	Encoder in listener mode	Appliance template expects listener; encoder caller

tsduckctl check may show bound (SRT session unverified) — that is normal without an active peer.

Ingest: SRT connected but `pixop-flow` quiet at :5500

Signals	Likely cause	Safe action
SRT app shows connected	No MPEG-TS payload or wrong encapsulation	Confirm encoder sends MPEG-TS over SRT; probe with partner tools
Relay logs, no TS	`--output stdout` missing or relay child crash	Validate template; `journalctl -u tsduck-in@main`
Exit 3 from `pixop-flow`	Input exceeds `TS_TARGET_BITRATE` at tap	Reduce source bitrate or adjust backbone per design

Processing: :5500 active, :5501 idle

Signals	Likely cause	Safe action
Cold start	GPU models loading	Keep continuous source ≥ 1 minute
`pixop-live` not running	Container fault	`pixopctl status`; `journalctl -u pixop-live`
License failure	Marketplace / JWT	`pixop-license doctor` or KVM license status

PID / picture: flow OK but wrong or frozen image

Signals	Likely cause	Safe action
Default PID 101 wrong for feed	PMT differs	READ-ONLY `tsduckctl detect-video-pid main --secs 10`
Unambiguous detection	—	APPLY only after review: `tsduckctl detect-video-pid main --apply`
PCR drift / continuity	PID/PCR mismatch	Re-run detect; avoid manual PMT hacks

Egress: bound on 5001 but receiver sees nothing

Signals	Likely cause	Safe action
`:5600` quiet in `pixop-flow`	Ingress/mux not delivering merged TS	Fix upstream layers first
Recent egress restart	`EGRESS_STARTUP_DRAIN_MS` discards initial TS	Wait ~2 s; retry receiver
Receiver passphrase/`pbkeylen`	Egress uses 32	Match `tsduckctl credentials`
SG on receiver path	UDP 5001 blocked	Open to receiver CIDR

Egress: processed video expected but output looks like source

Signals	Likely cause	Safe action
Early timeline	Pre-enhancement buffer in capture	Probe longer; select HEVC program in analyzer
Wrong output PID in monitor	Monitoring pre-mux tap	Measure at SRT egress, not encoder return feed

Multicast / PCR internal (operator awareness)

Multicast loss on 239.0.0.1 is almost always a local pipeline issue, not external switching. Symptoms mirror PID or PIPELINE_LATENCY stress:

Signals	Likely cause	Safe action
`:5501` drops under load	Latency too low for GPU path	READ-ONLY `tsduckctl tune-latency main`; APPLY after review
Continuity errors on `:5600`	PCR remap starvation	Tune latency; check `pixop-flow` exit 3

Relay ingress stall / blackhole (relay backend)

Signals	Likely cause	Safe action
Log lines: input reset, receive-queue-full	Path blackhole with prior media	Correlate with network event; recovery may be automatic
Idle source	No delivered payload	Recovery must not trigger — if it does, capture journals
Frequent resets	Encoder or path instability	Fix network; do not disable recovery without cause analysis

Wrong SRT key or mode

Signals	Likely cause	Safe action
libsrt encryption error	Passphrase typo	Re-export from `tsduckctl credentials` through secure channel
`pbkeylen=16` vs `32` swapped	Ingest vs egress confusion	Ingest 16, egress 32 per template
Native vs relay mismatch	Edited `.env` but relay template still old URI	When helper installed: `tsduckctl transport` warnings; align template and `TRANSPORT_BACKEND`

AWS security group / NACL

Signals	Likely cause	Safe action
`ss` shows `0.0.0.0:5000` on guest; peer fails	SG inbound	Add `SrtInputCidr` / `SrtOutputCidr` per AWS Marketplace quickstart
`pixop-preflight` OK externally broken	Preflight cannot see SG	Fix VPC rules; test with `srt-live-transmit` from allowed host
Outbound license errors	Egress to AWS blocked	Sample CFN allows outbound all — tighten only with care

KVM bridge vs userspace

Symptom	Context	Action
Cannot reach guest from LAN	Userspace install	Use `127.0.0.1` forwards or reinstall with `--bridge`
Works on host, not remote	SLIRP binding	Expected; bridge for production

Credential and file sensitivity

Risk	Mitigation
Passphrase in ticket	Use `tsduckctl credentials` on appliance only
World-readable `.env`	Must be `0600` root
Pasting secrets into `*.args`	Use `@SRT_PASSPHRASE@` only

Stream presence tools

Tool	Use
`pixop-flow --secs 5`	Mbps per stage; exit 0/3/10
`pixop-preflight`	Local listeners, license, disk — not SG
`tsduckctl check main`	Bind + unit active; may print `bound (SRT session unverified)`
`tsduckctl transport main`	Effective backend summary when helper installed (native or relay); extra relay bind detail when backend is relay
`curl -sS http://<management-host>:8080/v1/metrics`	Frame counters after input seen (restrict 8080 to `ManagementCidr` / trusted LAN)

Pixop does not expose a first-class “SRT packet loss percent” metric in these CLIs — do not infer loss from absence of such a field.

Command cheat sheet

Goal	Command	Mutates?
Listener readiness	`tsduckctl check main`	No
Effective transport	`tsduckctl transport main` (if `tsduck-transport-info` present)	No
UDP bind detail	`ss -uln` on ports 5000/5001 (any release)	No
Path bitrate	`pixop-flow --secs 5`	No
Video PID sample	`tsduckctl detect-video-pid main`	No
Apply PID	`tsduckctl detect-video-pid main --apply`	Yes — restarts ingress
Latency recommendation	`tsduckctl tune-latency main`	No
Apply latency	`tsduckctl tune-latency main --apply`	Yes — restarts ingress
Relay logs	`journalctl -u tsduck-in@main -n 200`	No
Multipath preflight	`pixop-net multipath verify --relay-args ...`	No

When to restart

Situation	Restart?
SG rule fixed	No — peers reconnect
Template or `.env` transport edit	Yes — affected `tsduckctl restart <instance>`
`detect-video-pid --apply` / `tune-latency --apply`	Automatic ingress restart
`pixopctl restart` after encoder change	No — for `nvencc.args` / Pixop only
Flapping SRT with no config change	No — diagnose peer/network first

Escalation evidence

When opening a support case, attach non-secret artifacts:

pixop-pack-config    # if support requests config-only bundle
pixop-support-bundle # when full diagnostics approved
pixop-version
tsduckctl check main
tsduckctl check egress-main
pixop-flow --secs 5

If command -v tsduck-transport-info succeeds, add tsduckctl transport main and tsduckctl transport egress-main.

Include: AMI/QCOW2 version, deployment (AWS/KVM), symptom layer (A–F above), timestamps of restarts, and whether relay or native backend is active. Never attach tsduckctl credentials output or .env passphrases to tickets.