i have redhat9.5 on RTX PRO6000 BlackwellServerEdition, i used nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml to generate nvidia.yaml for podman, but when i run podman with –device nvidia.com/gpu=all it failed , without –device nvidia.com/gpu=all it’s ok, anyone know this ?
with almost same config, h200 has no such issue
============
rpm -qa | grep nvidia-container
nvidia-container-runtime-3.14.0-1.noarch
nvidia-container-toolkit-base-1.17.8-1.x86_64
nvidia-container-toolkit-1.17.8-1.x86_64
libnvidia-container-tools-1.17.8-1.x86_64
libnvidia-container1-1.17.8-1.x86_64
podman --version
podman version 5.2.2
podman run -it --device nvidia.com/gpu=all xxxxxxx /bin/bash
Error: container create failed (no logs from conmon): conmon bytes “”: readObjectStart: expect { or n, but found , error found in #0 byte of …||…, bigger context …||…
with –log-level=debug
DEBU\[0000\] /usr/bin/conmon messages will be logged to syslog
DEBU\[0000\] running conmon: /usr/bin/conmon args=“\[–api-version 1 -c b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11 -u b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11/userdata -p /run/containers/storage/overlay-containers/b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11/userdata/pidfile -n lucid_dewdney --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11 --full-attach -l journald --log-level debug --syslog -t --conmon-pidfile /run/containers/storage/overlay-containers/b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11\]”
DEBU\[0000\] Cleaning up container b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11
DEBU\[0000\] Tearing down network namespace at /run/netns/netns-61f50418-575a-9331-29ec-ea9ebf9414dd for container b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11
\[DEBUG netavark::commands::teardown\] “Tearing down..”
\[INFO netavark::firewall\] Using iptables firewall driver
\[INFO netavark::network::bridge\] removing bridge podman0
\[DEBUG netavark::commands::teardown\] “Teardown complete”
DEBU\[0000\] Unmounted container “b2ba7c552b7c27dcab631e56aa95bc54b8c4f25f1ce76e91611f350687067d11”
DEBU\[0000\] ExitCode msg: “container create failed (no logs from conmon): conmon bytes "": readobjectstart: expect { or n, but found \\x00, error found in #0 byte of …||…, bigger context …||…”
Error: container create failed (no logs from conmon): conmon bytes “”: readObjectStart: expect { or n, but found , error found in #0 byte of …||…, bigger context …||…
DEBU\[0000\] Shutting down engines