I do not yet have a Thor to look at, but if you can log in to the Thor with a serial console before things go bad, and run the command “dmesg --follow”, then the PC which monitors the serial console can save a log after everything has crashed (serial console essentially logs to the alternate computer, and the other computer isn’t failed). You can also set the serial console to log at the start of the session so everything it sees prior to locking up should be available by posting the serial console log. Hopefully the log shows a stack frame.
You might also mention if there is anything known to be going on with networking at the time, e.g., a web browser is running, a software update or download is running, a multimedia stream is running, so on. Prior to things going bad, while serial console is logging, you might also the command “ip -s addr” and “ip route” so the logs see this prior to starting “dmesg --follow”.
At this point, I haven’t connected via serial console yet — the dmesg --follow output below was captured directly from the terminal while the system was still responsive. Here’s a snippet of what I see repeatedly:
These messages continue in a loop, and after some time, the system becomes unresponsive. This behavior is the same whether the system is connected to the internet or completely offline — no browser, no downloads, no updates.
I will try to capture the same output from a serial console next, as you suggested, to ensure logs are preserved even after the system hangs. If you have any insights based on the current log or thoughts about the repeated PCS block lock and nvAssertFailed messages, I’d appreciate your input.
Since this code is triggered by the unifiedgpudisp driver, I suspected a graphical desktop issue and switched the system to text-only mode. However, the lag persisted in text-only mode. The relevant log is attached.
I don’t have internal information on the Ethernet, but it kind of looks like it might be a race condition or just insufficient device tree setting. I see lots of this or related:
[ 9.571570] -->macsec_probe()
[ 9.572428] nvethernet a808a10000.ethernet: DT info about vlan in clear is missing setting default-disabled
[ 9.572436] -->macsec_get_platform_res()
[ 9.580266] <--macsec_get_platform_res()
[ 9.580273] -->macsec_enable_car()
[ 9.582170] <--macsec_enable_car()
[ 9.582199] <--macsec_probe()
[ 9.582202] nvethernet a808a10000.ethernet: Macsec: Reduced MTU: 1466 Max: 9000
[ 9.627268] nvethernet a808a10000.ethernet: mgbe0_0 (HW ver: 42) created with 4 DMA channels
[ 9.644053] nvethernet a808b10000.ethernet: Adding to iommu group 32
[ 9.647016] nvethernet a808b10000.ethernet: Virtualization is not enabled
[ 9.647025] nvethernet a808b10000.ethernet: failed to read skip mac reset flag, default 0
[ 9.647029] nvethernet a808b10000.ethernet: failed to read MDIO address
[ 9.647038] nvethernet a808b10000.ethernet: Failed to read nvida,pause_frames, so setting to default support as disable
[ 9.647040] nvethernet a808b10000.ethernet: Failed to read nvida,disable-rx-checksum, so setting to default - rx checksum offload enabled
[ 9.647044] nvethernet a808b10000.ethernet: setting to default DMA bit mask
[ 9.660545] nvethernet a808b10000.ethernet: failed to read or invalid MDC CR - default to 5
[ 9.660696] nvethernet a808b10000.ethernet: failed to get phy reset gpio error: -2
[ 9.672493] nvethernet a808b10000.ethernet: Ethernet MAC address: 3c:6d:66:e3:fb:d0
[ 9.672781] nvethernet a808b10000.ethernet: VM IRQ is handled by Camera CPU: 4
This isn’t clearly a particular cause, but failing some of the information related to the above could conceivably be a device tree issue. This goes on to look like maybe it is retrying “fixing” the issue over and over at different device tree physical address style locations (this could be completely unrelated, but it is suspicious).
For reference, here is the actual kernel stack frame and some content just prior to the stack frame:
Someone from NVIDIA can likely go straight to that code and get an idea of what is failing. Still, it would be useful to know, have you done any apt type update? Has anything at all been customized yet?
How to solve the “NVRM: nvAssertFailed: Assertion failed: 0 @ g_kern_bus_nvoc.h:2706” issue. I use nvidia/pytorch:25.08-py3 docker image, and run the gemma3 27B by HF transformers, the generated contents were in an uncertain state, sometimes generating garbage texts.
Update: I discovered a strange temporary approach to solve this issue:
Run ‘nvidia-smi dmon’ in the terminal
Then use the transformers python to inference the gemma27B, every time the generation was correct, and no garbage texts were output. Also, the ‘nvAssertFailed: Assertion failed’ error has gone.
same problem, and I find switch to wayland desktop solved also, another problem is the memory can’t be release when the ollama model is close, and the process can’t find any process, just memory still in there, I think the driver have many problem
The nvAssertFailed is still present on the up to date version of Jetpack7 r38.2-08-22.
What tricks can be used to minimize the issue ? Is there any patch for the kernel ?
Same issue here - system inputs e.g. mouse and keys and outputs, e.g., monitor, is stalling randomly. Jetpack7: R38 (release), REVISION: 2.2, GCID: 42205042, BOARD: generic, EABI: aarch64, DATE: Thu Sep 25 22:47:11 UTC 2025 # KERNEL_VARIANT: oot
I have updated to Jetpack 7.1 r38.4 but I am as well having trouble with unbearable ssh connection freezes and interruptions. These are the logs I could get using dmesg.
What even does this mean?
”nvethernet a808a10000.ethernet: [xpcs_lane_bring_up][827][type:0x4][loga-0x0] PCS block lock SUCCESS” and why is it logged that often?
[ 9.120870] /bus@0/aconnect@9000000/ahub@9630000/sfc@9730000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000
[ 9.121918] /bus@0/aconnect@9000000/ahub@9630000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000/processing-engine@9780000
[ 9.121934] /bus@0/aconnect@9000000/ahub@9630000/processing-engine@9780000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000
[ 9.122176] /bus@0/aconnect@9000000/ahub@9630000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000/mvc@9800000
[ 9.122191] /bus@0/aconnect@9000000/ahub@9630000/mvc@9800000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000
[ 9.123283] /bus@0/aconnect@9000000/ahub@9630000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000/mvc@9810000
[ 9.123297] /bus@0/aconnect@9000000/ahub@9630000/mvc@9810000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000
[ 9.123531] /bus@0/aconnect@9000000/ahub@9630000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000/amixer@9820000
[ 9.123544] /bus@0/aconnect@9000000/ahub@9630000/amixer@9820000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000
[ 9.123787] /bus@0/aconnect@9000000/ahub@9630000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000/asrc@9850000
[ 9.123807] /bus@0/aconnect@9000000/ahub@9630000/asrc@9850000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000
[ 9.124064] /bus@0/aconnect@9000000/ahub@9630000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000/arad@9870000
[ 9.124085] /bus@0/aconnect@9000000/ahub@9630000/arad@9870000: Fixed dependency cycle(s) with /bus@0/aconnect@9000000/ahub@9630000
[ 9.129127] tegra-se 8188140000.crypto: registered HASH algorithms
[ 9.131686] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx1: Adding to iommu group 48
[ 9.131734] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx1: initialized (streamid=4355)
[ 9.134094] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx2: Adding to iommu group 49
[ 9.134625] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx2: initialized (streamid=4356)
[ 9.136638] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx3: Adding to iommu group 50
[ 9.136683] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx3: initialized (streamid=4357)
[ 9.138411] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx4: Adding to iommu group 51
[ 9.140085] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx4: initialized (streamid=4358)
[ 9.140541] <–macsec_probe()
[ 9.187833] usbcore: registered new interface driver r8153_ecm
[ 9.197484] nvethernet a808d10000.ethernet: Macsec: Reduced MTU: 1466 Max: 9000
[ 9.218849] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx5: Adding to iommu group 52
[ 9.226316] nvethernet a808d10000.ethernet: mgbe2_0 (HW ver: 42) created with 4 DMA channels
[ 9.236693] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx5: initialized (streamid=4359)
[ 9.299353] r8152 2-3.1:1.0 enx00e04c080424: renamed from eth0
[ 9.318335] nvethernet a808e10000.ethernet: Adding to iommu group 53
[ 9.321010] eric-tx CALL alloc_txring !!!
[ 9.540412] rtk_btusb: download_data done
[ 9.541857] nvethernet a808e10000.ethernet: Virtualization is not enabled
[ 9.548787] rtk_btusb: HCI reset.
[ 9.548826] tegra210-admaif 9610000.admaif: Adding to iommu group 54
[ 9.554425] nvethernet a808e10000.ethernet: failed to read skip mac reset flag, default 0
[ 9.561446] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx6: Adding to iommu group 55
[ 9.562891] rtk_btusb: read_ver_rsp->lmp_subver = 0xcb71
[ 9.562893] rtk_btusb: read_ver_rsp->hci_rev = 0x40b
[ 9.562895] rtk_btusb: patch_entry->lmp_sub = 0x8852
[ 9.562900] rtk_btusb: Rtk patch end 0
[ 9.562901] rtk_btusb: chip type value: 0x78
[ 9.562902] rtk_btusb: btusb_open set HCI UP RUNNING
[ 9.562920] rtk_btcoex: Open BTCOEX
[ 9.562921] rtk_btusb: btusb_open end
[ 9.563902] rtk_btusb: ISO handle range (handle >= 001b)
[ 9.565547] nvethernet a808e10000.ethernet: failed to read MDIO address
[ 9.565550] nvethernet a808e10000.ethernet: Failed to read nvida,pause_frames, so setting to default support as disable
[ 9.565552] nvethernet a808e10000.ethernet: Failed to read nvida,disable-rx-checksum, so setting to default - rx checksum offload enabled
[ 9.565553] nvethernet a808e10000.ethernet: setting to default DMA bit mask
[ 9.568194] nvethernet a808e10000.ethernet: failed to read or invalid MDC CR - default to 5
[ 9.572806] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx6: initialized (streamid=4360)
[ 9.578676] rtl8852ce 0001:01:00.0 wlP1p1s0: renamed from wlan0
[ 9.579897] rtk_btcoex: BTCOEX hci_rev 0x040b
[ 9.579899] rtk_btcoex: BTCOEX lmp_subver 0xcb71
[ 9.592744] rtk_btcoex: rtk_vendor_cmd_to_fw: opcode 0xfc1b
[ 9.614360] nvethernet a808e10000.ethernet: failed to get phy reset gpio error: -2
[ 9.620937] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx7: Adding to iommu group 56
[ 9.624888] rtk_btusb: btusb_notify: hci0 evt 3
[ 9.630355] nvethernet a808e10000.ethernet: Ethernet MAC address: 4c:bb:47:0d:9b:7b
[ 9.630895] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx7: initialized (streamid=4361)
[ 9.637943] nvethernet a808e10000.ethernet: VM IRQ is handled by Camera CPU: 4
[ 9.645216] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx8: Adding to iommu group 57
[ 9.653244] input: NVIDIA Jetson Thor AGX APE Headset Jack as /devices/platform/sound/sound/card2/input5
[ 9.659884] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx8: initialized (streamid=4353)
[ 9.668027] tegra-audio-graph-card sound: Registered APE graph sound card with DPCM links for AHUB
[ 9.669100] -->macsec_probe()
[ 9.669122] nvethernet a808e10000.ethernet: DT info about vlan in clear is missing setting default-disabled
[ 9.669124] -->macsec_get_platform_res()
[ 9.669158] <–macsec_get_platform_res()
[ 9.669158] -->macsec_enable_car()
[ 9.669310] <–macsec_enable_car()
[ 9.669318] <–macsec_probe()
[ 9.669318] nvethernet a808e10000.ethernet: Macsec: Reduced MTU: 1466 Max: 9000
[ 9.670008] nvethernet a808e10000.ethernet: mgbe3_0 (HW ver: 42) created with 4 DMA channels
[ 9.674257] pva_kmd 818c000000.pva0: syncpt_unit_base 81c0000000 syncpt_unit_size 4000000 size 10000
[ 9.693531] camrtc-coe tegra-capture-coe0: netdev event 5 dev mgbe0_0
[ 9.700072] pva_iommu_context_dev 818c000000.pva0:pva0_niso1_ctx8: PVA syncpt (RO & RW) iova: 78000000, size: 4000000
[ 9.708764] camrtc-coe tegra-capture-coe0: Ch4->PDMA4
[ 9.710532] Allocated pva->fw_carveout
[ 9.720988] camrtc-coe tegra-capture-coe0: Ch5->PDMA5
[ 9.724474] pva_kmd 818c000000.pva0: PVA probe completed successfully
[ 9.737046] camrtc-coe tegra-capture-coe0: Ch6->PDMA6
[ 9.737280] tegra-mixer-controls sound:mixer-controls: Begin probe of override control device
[ 9.740771] nvpva_init completed: 0. GSC boot: 1
[ 9.747527] camrtc-coe tegra-capture-coe0: Ch7->PDMA7
[ 9.774704] tegra-mixer-controls sound:mixer-controls: Registered override controls for APE sound card
[ 9.777499] camrtc-coe tegra-capture-coe0: Ch8->PDMA7
[ 11.216727] camrtc-coe tegra-capture-coe0: Camera Over Eth controller a808a10000.ethernet num_chans=5 IRQ=4
[ 11.217478] camrtc-coe tegra-capture-coe1: netdev event 5 dev mgbe1_0
[ 11.221701] camrtc-coe tegra-capture-coe1: Ch4->PDMA4
[ 11.221743] camrtc-coe tegra-capture-coe1: Ch5->PDMA5
[ 11.221768] camrtc-coe tegra-capture-coe1: Ch6->PDMA6
[ 11.221786] camrtc-coe tegra-capture-coe1: Ch7->PDMA7
[ 11.221807] camrtc-coe tegra-capture-coe1: Ch8->PDMA7
[ 11.221872] camrtc-coe tegra-capture-coe1: Camera Over Eth controller a808b10000.ethernet num_chans=5 IRQ=4
[ 11.221965] camrtc-coe tegra-capture-coe2: netdev event 5 dev mgbe2_0
[ 11.222008] camrtc-coe tegra-capture-coe2: Ch4->PDMA4
[ 11.222031] camrtc-coe tegra-capture-coe2: Ch5->PDMA5
[ 11.222050] camrtc-coe tegra-capture-coe2: Ch6->PDMA6
[ 11.222074] camrtc-coe tegra-capture-coe2: Ch7->PDMA7
[ 11.222097] camrtc-coe tegra-capture-coe2: Ch8->PDMA7
[ 11.222143] camrtc-coe tegra-capture-coe2: Camera Over Eth controller a808d10000.ethernet num_chans=5 IRQ=4
[ 11.222198] camrtc-coe tegra-capture-coe3: netdev event 5 dev mgbe3_0
[ 11.222226] camrtc-coe tegra-capture-coe3: Ch4->PDMA4
[ 11.222244] camrtc-coe tegra-capture-coe3: Ch5->PDMA5
[ 11.222267] camrtc-coe tegra-capture-coe3: Ch6->PDMA6
[ 11.222291] camrtc-coe tegra-capture-coe3: Ch7->PDMA7
[ 11.222310] camrtc-coe tegra-capture-coe3: Ch8->PDMA7
[ 11.222352] camrtc-coe tegra-capture-coe3: Camera Over Eth controller a808e10000.ethernet num_chans=5 IRQ=4
[ 11.408304] audit: type=1400 audit(1768825602.960:2): apparmor=“STATUS” operation=“profile_load” profile=“unconfined” name=“busybox” pid=1009 comm=“apparmor_parser”
[ 11.409560] audit: type=1400 audit(1768825602.960:3): apparmor=“STATUS” operation=“profile_load” profile=“unconfined” name=4D6F6E676F444220436F6D70617373 pid=1004 comm=“apparmor_parser”
[ 11.423758] audit: audit_backlog=65 > audit_backlog_limit=64
[ 11.426760] audit: type=1400 audit(1768825602.960:4): apparmor=“STATUS” operation=“profile_load” profile=“unconfined” name=“buildah” pid=1008 comm=“apparmor_parser”
[ 11.431518] audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=64
[ 11.446137] audit: type=1400 audit(1768825602.964:5): apparmor=“STATUS” operation=“profile_load” profile=“unconfined” name=“vscode” pid=1014 comm=“apparmor_parser”
[ 11.453259] audit: backlog limit exceeded
[ 11.467732] audit: type=1400 audit(1768825602.964:6): apparmor=“STATUS” operation=“profile_load” profile=“unconfined” name=“desktop-icons-ng” pid=1019 comm=“apparmor_parser”
[ 11.471813] audit: audit_backlog=65 > audit_backlog_limit=64
[ 11.487247] audit: type=1400 audit(1768825602.964:7): apparmor=“STATUS” operation=“profile_load” profile=“unconfined” name=“QtWebEngineProcess” pid=1005 comm=“apparmor_parser”
[ 11.531066] enP2p1s0: 0xffff800088dc0000, 4c:bb:47:13:c3:01, IRQ 288
[ 11.542028] r8126 0002:01:00.0 enP2p1s0: registered PHC device on enP2p1s0
[ 11.542032] r8126 0002:01:00.0 enP2p1s0: reset PHC clock
[ 11.730034] rtk_btusb: btusb_flush add delay
[ 11.740045] rtk_btusb: btusb_close
[ 11.750919] rtk_btcoex: Close BTCOEX
[ 11.750922] rtk_btcoex: -x
[ 12.071605] nv_nvsciipc_init.sh[1215]: found /dev/nvsciipc
[ 12.075824] nvsciipc: nvsciipc_uid is set as 2000
[ 12.075858] nv_nvsciipc_init.sh[1215]: set 2000 to /sys/devices/platform/nvsciipc/nvsciipc_uid
[ 12.094680] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 12.094685] Bluetooth: BNEP filters: protocol multicast
[ 12.094689] Bluetooth: BNEP socket layer initialized
[ 12.096602] Bluetooth: MGMT ver 1.22
[ 12.099356] NET: Registered PF_ALG protocol family
[ 12.101315] loop16: detected capacity change from 0 to 32768
[ 12.105491] rtk_btusb: btusb_open start
[ 12.105496] rtk_btusb: btusb_open hdev->promisc ==0
[ 12.105496] rtk_btusb: download_patch start
[ 12.105498] rtk_btusb: chip type value: 0x78
[ 12.106897] rtk_btusb: chip_type->status = 0x0, chip_type->chip = 0xbeef
[ 12.106902] rtk_btusb: HCI reset.
[ 12.109009] nvsciipc: set_db start
[ 12.109031] nvsciipc: set_db done
[ 12.114223] nvidia-nvlink: Nvlink Core is being initialized, major device number 490
Hi,
For now we don’t observe this on AGX Thor developer kit with Jetpack 7.1GA. Please check if you can reproduce it on developer kit and share us the one-by-one steps. We will set up developer kit and check.
