Hi, I’m trying to deploy the Isaac Sim container on a SLURM compute cluster with A6000 GPUs and access it through the omniverse native client. The drivers version is 555. Since only podman containers are allowed on the cluster and the nvidia-container-toolkit cannot be installed, so far I’ve managed to get other compute workloads to work by passing the required libraries and devices through to the container and creating symlinks where needed. However, when I run Isaac Sim in headless mode in my podman container like this:
./runheadless.native.sh --/app/livestream/logLevel=verbose
the streaming server fails to start with this error:
2024-07-26 10:53:18 [8,234ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #8(M)[2024-07-26 10:53:18,059]=10:53:18=*00000*{4475518564059289201}&176802
1A&<VersionDump> Name: unnamed
2024-07-26 10:53:18 [8,234ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #9(M)[2024-07-26 10:53:18,059]=10:53:18=*00000*{4475518564059289201}&CF6CAAC5&<VersionDump> DVS Branch: gcomp/rel/gs_04
_59
2024-07-26 10:53:18 [8,234ms] [Fatal] [carb.livestream.plugin] StreamSDK: VersionDump: #0(M)[2024-07-26 10:53:18,059]=10:53:18=*00000*{4475518564059289201}&EA649458&<VersionDump> DVS P4VirtualChangelist: 33
332890
2024-07-26 10:53:18 [8,234ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #3(M)[2024-07-26 10:53:18,059]=10:53:18=*00000*{4475518564059289201}&7DF5E4E1&<Server> Using: OpenSSL 1.1.1q 5 Jul 202
2
2024-07-26 10:53:18 [8,234ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #4(M)[2024-07-26 10:53:18,059]=10:53:18=*00000*{4475518564059289201}&55B93E2E&<Server> RND is initialized
2024-07-26 10:53:18 [8,235ms] [Error] [carb.livestream.plugin] StreamSDK: NvNetworkCommon: #1(E)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&168334B0&<NvNetworkCommon> Network Address To Stri
ng: invalid parameters 0x7f1070bfd9a0, 0x7f1070bfd9c0, 48
2024-07-26 10:53:18 [8,235ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #5(M)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&AD28CD28&<Server> --- B8:CE:F6:A1:D3:5A: 10.126.16
4.8 / invalid
2024-07-26 10:53:18 [8,235ms] [Error] [carb.livestream.plugin] StreamSDK: NvNetworkCommon: #0(E)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&168334B0&<NvNetworkCommon> Network Address To Stri
ng: invalid parameters 0x7f1070bfd9a0, 0x7f1070bfd9c0, 48
2024-07-26 10:53:18 [8,235ms] [Fatal] [carb.livestream.plugin] StreamSDK: Server: #4(M)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&AD28CD28&<Server> --- B8:CE:F6:A1:D3:5A: 10.126.16
4.8 / invalid
2024-07-26 10:53:18 [8,236ms] [Fatal] [carb.livestream.plugin] StreamSDK: NvEncUtil: #0(M)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&CC7C5461&<NvEncUtil> Computing SPS/PPS headers.
2024-07-26 10:53:18 [8,236ms] [Warning] [carb.livestream.plugin] StreamSDK: VideoFrameCtxt: #1(W)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&0564BB28&<VideoFrameCtxt> Couldn't find VideoFra
meContext for this thread.
2024-07-26 10:53:18 [8,236ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #3(M)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&1E959A86&<AdapterContext> Initializing Adapter con
text
2024-07-26 10:53:18 [8,236ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #6(M)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&F0E847F4&<AdapterContext> Initialized context for
adapter 0: 1280 x 720 @ 60.0 Hz
2024-07-26 10:53:18 [8,236ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #8(M)[2024-07-26 10:53:18,060]=10:53:18=*00000*{4475518564059289201}&9A64D99C&<AdapterContext> System is NOT co-proc.
2024-07-26 10:53:18 [8,385ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #9(M)[2024-07-26 10:53:18,209]=10:53:18=*00148*{4475518564059289201}&21C9B327&<AdapterContext> GPU memory usage: used =
643.437500, free = 44879.500000 MB, total = 45522.937500 MB
2024-07-26 10:53:18 [8,385ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #0(M)[2024-07-26 10:53:18,209]=10:53:18=*00000*{4475518564059289201}&DD40CDEB&<AdapterContext> Initialized CUDA for dev
ice 'NVIDIA RTX A6000' (SM 8.6) in compute mode 'CU_COMPUTEMODE_DEFAULT'.
2024-07-26 10:53:18 [8,385ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #1(E)[2024-07-26 10:53:18,210]=10:53:18=*00000*{4475518564059289201}&5FA2F6E4&<NvEncUtil> Failed to allocate encoder for RTSP SPS/PPS header generation
2024-07-26 10:53:18 [8,385ms] [Fatal] [carb.livestream.plugin] StreamSDK: AdapterContext: #2(M)[2024-07-26 10:53:18,210]=10:53:18=*00000*{4475518564059289201}&21C9B327&<AdapterContext> GPU memory usage: used =
643.437500, free = 44879.500000 MB, total = 45522.937500 MB
2024-07-26 10:53:18 [8,473ms] [Error] [carb.livestream.plugin] StreamSDK: NvEncUtil: #5(E)[2024-07-26 10:53:18,298]=10:53:18=*00000*{4475518564059289201}&099DD006&<NvEncUtil> Error computing SPS/PPS heade
rs.
specifically it seems to fail here:
Failed to allocate encoder for RTSP SPS/PPS header generation
but the application doesn’t provide any additional information on why that fails. How can I get some more insights? I think it’s failing to find some library or file, but like this I have no idea of what it could be