pixop-startup-timeline

Collect cold-start milestones from existing systemd timestamps, journald excerpts, and /var/log/pixop-live/live.log pattern matches. Used to separate slow first-boot GPU/model paths from EBS volume hydration on AWS.

Availability: Verify this tool on your appliance with command -v pixop-startup-timeline and pixop-version (may be absent on some recipes). Installed path when present: /opt/pixop/bin/pixop-startup-timeline.

Purpose and scope

Read-only: does not warm disks, change services, enable profiling, or print secrets. Complements journalctl and live.log with a single scrollable report for support.

Non-root users are re-execed via sudo.

Usage

pixop-startup-timeline

No flags. Optional environment:

Variable	Default
`PIXOP_LIVE_LOG`	`/var/log/pixop-live/live.log`

Output sections

Host boot

systemd-analyze summary when available.
journald start timestamp from systemd-journald.service inactive exit time.

systemd units

For each unit, prints ActiveEnterTimestamp and ExecMainStartTimestamp:

docker.service
nvidia-persistenced.service
pixop-instance-init.service
pixop-live.service
[email protected]
[email protected]
pixop-healthcheck.timer

Use these to see ordering gaps (for example Docker before live, init before ingress).

Journald lifecycle (this boot)

Filtered journalctl -b lines (up to 80) for the units above matching start/finish/license/error patterns.

Pixop live.log milestones

Scans live.log for first matching lines (most recent boot wins if log was appended):

Milestone key	Typical signal
`wrapper_license_warn`	License file not found (early boot)
`nvencc_banner`	NVEncC banner
`cuda_api`	NVENC API init
`gpu_device`	GPU enumeration
`nvencc_launch`	Launching `/opt/pixop/nvencc`
`rest_http`	REST HTTP server started
`model_begin` / `model_ready`	Model load phases
`first_frame`	First decoded/encoded frame style messages

n/a ... not found means the pattern never appeared (truncated log, failed start, or different log format).

Current listeners

UDP ports 5000–5003 from ss, or a note if none bound.

Closing reminder: timeline reads existing logs only; it does not warm EBS or start services.

Interpreting results

Pattern	Likely interpretation
Long gap `nvencc_launch` → `rest_http`	GPU/driver or container bring-up
Long gap `model_begin` → `model_ready`	Model load from image/weights (first launch can be minutes)
Long gap before any `live.log` line while units active	Possible EBS snapshot hydration on fresh AWS volumes — not fixed by this tool
`instance-init` late vs ingress	First-boot credential generation still running
Missing UDP 5000 at end	Ingress not up yet or stack stopped

Compare two boots: one immediately after launch, one after the instance has been up — hydration delays often shrink on the second boot.

Operator workflows

After a slow first Marketplace launch:

pixop-startup-timeline | tee /tmp/startup-timeline.txt
pixop-version
pixop-preflight | tail -n 5

KVM cold boot diagnosis:

sudo reboot
# after SSH returns
pixop-startup-timeline

Pair with pixop-support-bundle if support needs full journals (timeline is a lightweight slice).

Limitations

Pattern matching on live.log is heuristic; upgraded Pixop LIVE images may change log text (not found does not alone prove failure).
Does not timestamp every subsystem (no per-container Docker events).
Does not replace GPU benchmarks or pixop-test-live.
Cannot accelerate EBS — see CloudFormation volume initialization notes in engineering cloudformation/README.md.

Privacy

No passphrases or JWT content. Journal snippets may contain instance identifiers or internal URLs — handle like other operational logs.

Monitoring and troubleshooting — EBS cold hydration
pixop-support-bundle
pixop-preflight
AWS Marketplace quickstart