KVM troubleshooting guide

KVM troubleshooting guide

Symptom-oriented triage for Pixop Prime on KVM. Run host checks first, then guest checks over SSH.

Host (Ubuntu KVM hypervisor) — from extracted installer root:

cd /path/to/extracted-installer
sudo ./scripts/pixop-install doctor
sudo ./scripts/pixop-install status

Guest (Amazon Linux 2023 appliance) — SSH as ec2-user:

pixop-preflight
pixop-stack status
pixopctl status
tsduckctl check main
tsduckctl check egress-main
pixop-flow

Pipeline issues (metrics, recovery, TSDuck) are also in Monitoring and troubleshooting.

VM does not start

SymptomLikely causeVerifyRemediation
Domain start fails immediatelyKVM permissions or IOMMUdoctor: kvm, iommu linesNew login after kvm group change; enable VT-d/AMD-Vi in firmware and kernel
GPU passthrough errorsDevice not VFIO-viable or in usedoctor GPU table; status driver columnFree GPU from host workloads; pick BDF doctor marks viable
libvirt errorsService downsudo virsh -c qemu:///system list --allStart libvirt; review domain log
ENOMEM / memlockQEMU cannot lock RAM for VFIOdoctor memlock lineRaise libvirt/QEMU memlock limit or lower guest --memory

status while running should show vfio-pci on assigned GPUs and audio functions.

VFIO_MAP_DMA: Cannot allocate memory

Pinned-memory failure—not the high-BAR case below. Domain memlock is guest RAM plus 1 GiB (VRAM not added). Increase the service memlock limit or reduce guest RAM, then stop/start the VM.

VFIO_MAP_DMA failed: Invalid argument (high BAR)

On some IOMMUs, Resizable BAR mapped high can log EINVAL while encode/decode still work. Peer-to-peer and GPUDirect are not qualified on this installer. Do not disable Resizable BAR only to silence the message.

GPU missing inside guest

  • doctor must show VFIO viable; integrated graphics is never passed through.
  • status with VM running: expect vfio-pci for GPU and audio function.
  • Host still shows nvidia driver: domain did not take the device—check sudo virsh -c qemu:///system domstate <domain> and libvirt logs.
  • --gpu all skips non-viable devices; use explicit --gpu only when doctor approves that BDF.

License rejected, expired, or unclear status

Do not reinstall the VM for a token change.

GUEST_IP="203.0.113.10"
cd /path/to/extracted-installer
sudo ./scripts/pixop-install license status \
  --identity ~/.ssh/id_ed25519 \
  --guest-host "${GUEST_IP}"
sudo ./scripts/pixop-install license set /path/to/license.jwt \
  --identity ~/.ssh/id_ed25519 \
  --guest-host "${GUEST_IP}"

Bridged installs require --guest-host. VM must be running. license status shows file hash immediately; validity log lines can lag up to ~1 hour because Pixop is not restarted on renewal. First install with no file yet: add --restart-if-needed when prompted.

On the guest: pixop-preflight (license warnings before JWT is applied are expected).

SRT connected but no enhanced output yet

Cold start may load restoration models before enhanced output appears. Use a continuous live source; short clips can end too early.

tsduckctl credentials
pixop-flow
curl -sS http://127.0.0.1:8080/v1/metrics

From the host (bridged guest):

GUEST_IP="203.0.113.10"
curl -sS "http://${GUEST_IP}:8080/v1/metrics"

tsduckctl credentials re-executes via sudo internally—do not prefix with sudo tsduckctl. Ingest UDP 5000, egress UDP 5001. Passphrases are not stored on the KVM host.

Confirm codec, resolution, frame rate, and HDR/SDR with your downstream analyzer and pixop-version—do not assume a fixed upscale factor or HDR profile without measuring output.

One CUDA error then service healthy

After first VFIO boot the driver may still be settling. systemd may restart pixop-live once; treat as failure only if the unit stays failed or GPUs drop off the bus.

After uninstall, host GPU missing

uninstall should rebind GPUs from vfio-pci to nvidia. If a GPU is still gone, check lspci -k and ensure no stale domain holds the device.

Logs to collect for support

LayerCommand / path
Host install statesudo ./scripts/pixop-install status (redact IPs if needed)
Guest Pixopjournalctl -u pixop-live, /var/log/pixop-live/live.log
Guest TSDucktsduckctl logs main, tsduckctl logs egress-main
Guest snapshotpixopctl health, pixop-flow --secs 5

Include pixop-version, checksum verification result, and license file hash from license status—not the JWT body.

Clean rollback

cd /path/to/extracted-installer
sudo ./scripts/pixop-install stop
sudo ./scripts/pixop-install uninstall
# or, only with explicit approval and backup:
sudo ./scripts/pixop-install uninstall --purge

Your original delivery QCOW2 file is untouched.

See also


Did this page help you?