We are just getting started with the Jetson Thor. After flashing Thor with the latest Jetson ISO (r38.2-08-22) we see an error in dmesg (see below).
lspci also shows errors
sudo lspci -vvvv -s 0000:01:00.0
Also the gpu information in /proc seems wrong.
cat /proc/driver/nvidia/gpus/0000:01:00.0/information
[ 13.219822] ------------[ cut here ]------------#1
Trying to run the container loopback test also has errors.
===================================================================================================== short test summary info ====================================================================================================== test_holoscan[720-1280-bayer_format0-pixel_format0-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format1-pixel_format1-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format2-pixel_format2-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format3-pixel_format3-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format4-pixel_format4-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format5-pixel_format5-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)6 failed , 16 passed, 141 skipped, 8 warnings in 4.05s =======================================================================================
Could someone please provide some guidance for debugging this ?
Thank you,
-John
Could you flash with rel-38.2.1 with sdkmanager?
Thanks @wushanbi19 , @john.miller reflashed with the latest from the sdkmanager; however, when he progressed to running the HSB demo container, things went south. Here is what John did…
OK, i flashed with rel-38.2.1 with the SDK, but still see the kernel panic. After a cold boot lspci reports OK for the gpu.
However after i run the following holoscan test commands the GPU control plane wedges.
$ git clone https://github.com/nvidia-holoscan/holoscan-sensor-bridge
cd holoscan-sensor-bridge
sh docker/build.sh --igpu
xhost +
sh docker/demo.sh
pytest
Here is the test output.
===================================================================================================== short test summary info ====================================================================================================== test_holoscan[720-1280-bayer_format0-pixel_format0-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format1-pixel_format1-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format2-pixel_format2-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format3-pixel_format3-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format4-pixel_format4-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)test_holoscan[720-1280-bayer_format5-pixel_format5-0.1] - cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)6 failed , 16 passed, 141 skipped, 8 warnings in 4.05s =======================================================================================
Inspecting dmesg shows a lot not happy. Our thinking is that HSB and Thor aren’t best-friends yet. Can you suggest what we try next?
Shep
please attach your full dmesg so that we can see what is wrong first.
Actually, the kernel log in first comment was not fatal. It does not matter to the error.
Wayne, John and I both tried to include the dmesg that shows the failure; however the forum software is picking up the many ampersands in the output and doesn’t let us post as it thinks they are users being referenced! Is there a way we can attach or email you?
shep
Just want to clarify. This forum could attach file up.
Generally we don’t want users to directly copy + paste a 100-lines text to forum directly.
Apologies for the length, but here is the requested full dmesg output of the failure, with ampersands replaced so as not to upset the forum software…
However after i run the following holoscan test commands the GPU control plane wedges.
$ git clone GitHub - nvidia-holoscan/holoscan-sensor-bridge: NVIDIA Holoscan Sensor Bridge - Bring Your Own Sensor (BYOS) over Ethernet
cd holoscan-sensor-bridge
Here is the test output.
===================================================================================================== short test summary info ======================================================================================================
And here are the errors in dmesg
[ 101.119653] irq 265: nobody cared (try booting with the “irqpoll” option)#1 #265
[Network events and other entries continue normally…]
Once the test completes the gpu is wedged.
sudo lspci -vvvv -s 0000:01:00.0
Hi,
I am not sure if you get my point or not.
I mean you could add text file into your gedit, save it as a file and then upload that file.
Also, I need your error log right after the error happened.
So to make sure we are on the same page. Please define what error you want to discuss here first.
There is exactly one error here: The HSB demo container example is failing with AGX Thor. The application error is posted earlier in the thread; as is the dmesg spew that is farther from the scene of the crime.
Sorry, I didn’t see the upload icon on the edit bar. Will use that going forward.
Ok, then please file new topic for those lspci error. We will only discuss the HSB issue in this topic.
The lspci is a side-effect, a consequence, of the HSB issue. Like the disturbing dmesg output, the faulty is lspci was caused-by running the HSB program as in the docs. Yes, lets only discuss the HSB issue please. … Apologies for any confusion.
I’ve not tried it but have you followed thor-jp7-setup.md ?
hello shepard.siegel,
let me double check how you setup an environment to test with HSB.
hello shepard.siegel,
shepard.siegel:
cupy.cuda.compiler.CompileException: nvrtc: error: invalid value for --gpu-architecture (-arch)
BTW,Leopard Eagle VB1940 cameras for testing HSB with AGX Thor on JP-7.0?
Jerry,
let’s using Topic 345720 for tracking demo container failure.