On the Thor DevKit, the wired network occasionally fails

Robotics & Edge Computing Jetson Systems Jetson Thor
1

I am using the standard DevKit.Checking the kernel logs reveals PCIe errors, and a reboot is the only way to restore functionality.

log as follow:

[ 3702.062436] pcieport 0002:00:00.0: DPC: containment event, status:0x3f01 source:0x0000
[ 3702.062446] pcieport 0002:00:00.0: AER: Correctable error message received from 0002:00:00.0
[ 3702.062452] pcieport 0002:00:00.0: DPC: unmasked uncorrectable error detected
[ 3702.062476] pcieport 0002:00:00.0: AER: found no error details for 0002:00:00.0
[ 3702.062496] pcieport 0002:00:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
[ 3702.067023] pcieport 0002:00:00.0: device [10de:22d8] error status/mask=00004000/04400000
[ 3702.075394] pcieport 0002:00:00.0: [14] CmpltTO (First)
[ 3702.082046] r8126 0002:01:00.0: AER: can’t recover (no error_detected callback)
[ 3702.082084] CPU:0, Error: top-cbb-fabric@0x8100800000, irq=16
[ 3702.087964] **************************************
[ 3702.092500] CPU:0, Error:top-cbb-fabric, Errmon:1
[ 3702.097393] Error Code : DECODE_ERR
[ 3702.101230] Overflow : Multiple DECODE_ERR

[ 3702.106818] Error Code : DECODE_ERR
[ 3702.110661] MASTER_ID : CCPLEX
[ 3702.114152] Address : 0x30000010
[ 3702.117645] Cache : 0x1 – Bufferable
[ 3702.121836] Protection : 0x2 – Unprivileged, Non-Secure, Data Access
[ 3702.128820] Access_Type : Read
[ 3702.131963] Access_ID : 0x6
[ 3702.131964] Fabric_Id : 0x4
[ 3702.138249] Fabric : uphy0-cbb-fabric
[ 3702.142443] or Fabric : aon-fabric
[ 3702.145933] Please use Address to determine correct fabric.
[ 3702.151870] Slave_Id : 0x23
[ 3702.155013] Burst_length : 0x0
[ 3702.158507] Burst_type : 0x1
[ 3702.161649] Beat_size : 0x2
[ 3702.164792] VQC : 0x0
[ 3702.167585] GRPSEC : 0x7e
[ 3702.170379] FALCONSEC : 0x0
[ 3702.173525] **************************************
[ 3702.178422] ------------[ cut here ]------------
[ 3702.178423] WARNING: CPU: 0 PID: 0 at drivers/soc/tegra/cbb/tegra234-cbb.c:648 tegra234_cbb_isr+0x150/0x170
[ 3702.178434] Modules linked in: rfcomm xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype nft_compat nf_tables libcrc32c nvidia_drm(O) nvidia_modeset(O) nvidia_uvm(O) qrtr bridge stp llc usb_f_ncm usb_f_mass_storage nvidia(O) usb_f_acm governor_pod_scaling(O) u_serial usb_f_rndis u_ether libcomposite algif_hash algif_skcipher af_alg bnep snd_soc_tegra_controls(O) snd_soc_tegra_utils(O) snd_soc_tegra210_admaif snd_soc_tegra186_arad(O) snd_soc_tegra210_mixer snd_soc_tegra210_ope snd_soc_tegra_pcm snd_soc_tegra210_mvc snd_soc_tegra210_sfc snd_soc_tegra186_asrc snd_soc_tegra210_adx snd_soc_tegra210_amx snd_soc_tegra210_i2s snd_soc_tegra210_ahub nvvrs_pseq_rtc(O) tegra210_adma spidev rtk_btusb(O) btusb btrtl btintel btmtk btbcm bluetooth ecdh_generic ecc rtl8852ce(O) tegra_capture_coe(O) crct10dif_ce sm3_ce sm3 sha3_ce sha512_ce cfg80211 sha512_arm64 r8126(O) rfkill ina3221 ina238 tegra_aconnect tegra23x_psc(O) tegra_cactmon_mc_all(O) snd_hda_codec_hdmi at24 snd_hda_tegra
[ 3702.178488] snd_hda_codec nvidia_vrs_pseq(O) snd_soc_rt5640 snd_hda_core snd_soc_rl6231 host1x_fence(O) mttcan(O) can_dev pwm_tegra_tachometer(O) lm90 nvmap(O) spi_tegra114 nvidia_cspmu nvsciipc(O) ivc_cdev(O) nvethernet(O) coresight_trbe snd_soc_tegra_audio_graph_card snd_soc_audio_graph_card coresight snd_soc_simple_card_utils arm_spe_pmu nvpps(O) nvpmodel_clk_cap(O) tegra234_oc_event(O) thermal_trip_event(O) tegra_dce(O) ramoops arm_cspmu_module reed_solomon nvhost_pva(O) nvhost_nvcsi(O) tegra_se(O) nvhost_vi5(O) tegra_se_kds(O) crypto_engine nvhost_capture(O) tpm_ftpm_tee camera_diagnostics(O) nvhost_isp5(O) tegra_capture_isp(O) tegra_camera(O) v4l2_dv_timings host1x_nvhost(O) tegra_drm(O) tegra_wmark(O) nvhwpm(O) drm_display_helper drm_dp_aux_bus cec drm_kms_helper host1x(O) tegra_camera_platform(O) mc_utils(O) capture_ivc(O) v4l2_fwnode v4l2_async videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videodev videobuf2_common mc camchar(O) rtcpu_debug(O) tegra_camera_rtcpu(O) ivc_bus(O) hsp_mailbox_client(O)
[ 3702.178535] dm_multipath nvme_fabrics fuse drm nfnetlink ip_tables x_tables ipv6 pwm_fan pwm_tegra tegra_bpmp_thermal tegra_xudc uas ucsi_ccg typec_ucsi typec nvme nvme_core phy_tegra194_p2u pcie_tegra194 ufs_tegra(O) pcie_tegra264(O)
[ 3702.178552] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 6.8.12-tegra #1
[ 3702.178554] Hardware name: NVIDIA NVIDIA Jetson AGX Thor Developer Kit/Jetson, BIOS 38.2.0-gcid-41844464 08/22/2025
[ 3702.178555] pstate: 634000c9 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=–)
[ 3702.178558] pc : tegra234_cbb_isr+0x150/0x170
[ 3702.178559] lr : tegra234_cbb_isr+0x120/0x170
[ 3702.178560] sp : ffff800080003ea0
[ 3702.178561] x29: ffff800080003ea0 x28: ffffd8756a0b3880 x27: 0000000000000001
[ 3702.178563] x26: 00000000000000c0 x25: ffffd875693d60c8 x24: ffffd8756a6cc158
[ 3702.178565] x23: ffffd87569772008 x22: 0000000000000010 x21: ffffd8756a4703d0
[ 3702.178566] x20: ffffd8756a470490 x19: 0000000000000000 x18: 0000000000000006
[ 3702.178568] x17: ffff27a9ee587000 x16: ffffd87567e75c74 x15: ffff8000800036f0
[ 3702.178569] x14: 0000000000000aef x13: 00000000ffffffea x12: ffffd8756a123e80
[ 3702.178571] x11: 0000000000000001 x10: 0000000000000001 x9 : 0000000000017fe8
[ 3702.178573] x8 : ffffd8756a0cbe08 x7 : c0000000ffffefff x6 : 0000000000057fa8
[ 3702.178575] x5 : ffff001f57cfbd08 x4 : 0000000000000000 x3 : 0000000000000000
[ 3702.178576] x2 : 0000000000000000 x1 : ffffd8756a0b3880 x0 : 0000000100010001
[ 3702.178578] Call trace:
[ 3702.178579] tegra234_cbb_isr+0x150/0x170
[ 3702.178581] __handle_irq_event_percpu+0x5c/0x244
[ 3702.178590] handle_irq_event+0x4c/0xb8
[ 3702.178592] handle_fasteoi_irq+0xa4/0x1c4
[ 3702.178595] handle_irq_desc+0x48/0x80
[ 3702.178597] generic_handle_domain_irq+0x1c/0x30
[ 3702.178600] gic_handle_irq+0x4c/0x12c
[ 3702.178602] call_on_irq_stack+0x24/0x4c
[ 3702.178607] do_interrupt_handler+0x88/0x9c
[ 3702.178610] el1_interrupt+0x34/0x70
[ 3702.178615] el1h_64_irq_handler+0x18/0x2c
[ 3702.178617] el1h_64_irq+0x68/0x6c
[ 3702.178618] cpuidle_enter_state+0xc0/0x4c4
[ 3702.178621] cpuidle_enter+0x38/0x58
[ 3702.178624] do_idle+0x1fc/0x278
[ 3702.178627] cpu_startup_entry+0x38/0x3c
[ 3702.178629] kernel_init+0x0/0x1ec
[ 3702.178632] arch_post_acpi_subsys_init+0x0/0x18
[ 3702.178637] start_kernel+0x54c/0x6f4
[ 3702.178639] __primary_switched+0xb8/0xc0
[ 3702.178643] —[ end trace 0000000000000000 ]—
[ 3702.312920] pcieport 0002:00:00.0: AER: device recovery failed
[ 3708.172813] r8126 0002:01:00.0 enP2p1s0: NETDEV WATCHDOG: CPU: 5: transmit queue 0 timed out 6008 ms
[ 3708.172840] r8126 0002:01:00.0 enP2p1s0: Transmit timeout reset Device!
[ 3708.196814] r8126 0002:01:00.0 enP2p1s0: Device reseting!
[ 3708.203949] pcieport 0002:00:00.0: AER: Correctable error message received from 0002:00:00.0
[ 3708.203951] pcieport 0002:00:00.0: DPC: containment event, status:0x3f01 source:0x0000
[ 3708.203956] pcieport 0002:00:00.0: DPC: unmasked uncorrectable error detected
[ 3708.203981] pcieport 0002:00:00.0: AER: found no error details for 0002:00:00.0
[ 3708.203998] pcieport 0002:00:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Requester ID)
[ 3708.208522] pcieport 0002:00:00.0: device [10de:22d8] error status/mask=00004000/04400000
[ 3708.215063] r8126 0002:01:00.0 enP2p1s0: reset PHC clock
[ 3708.216897] pcieport 0002:00:00.0: [14] CmpltTO (First)
[ 3708.216918] r8126 0002:01:00.0: AER: can’t recover (no error_detected callback)
[ 3708.223557] CPU:0, Error: top-cbb-fabric@0x8100800000, irq=16
[ 3708.229464] **************************************
[ 3708.234003] CPU:0, Error:top-cbb-fabric, Errmon:1
[ 3708.238894] Error Code : DECODE_ERR
[ 3708.242733] Overflow : Multiple DECODE_ERR

[ 3708.248321] Error Code : DECODE_ERR
[ 3708.252163] MASTER_ID : CCPLEX
[ 3708.255655] Address : 0x300000b8
[ 3708.259147] Cache : 0x1 – Bufferable
[ 3708.263338] Protection : 0x2 – Unprivileged, Non-Secure, Data Access
[ 3708.270323] Access_Type : Read
[ 3708.273466] Access_ID : 0x5
[ 3708.273467] Fabric_Id : 0x4
[ 3708.279752] Fabric : uphy0-cbb-fabric
[ 3708.283945] or Fabric : aon-fabric
[ 3708.287435] Please use Address to determine correct fabric.
[ 3708.293372] Slave_Id : 0x23
[ 3708.296515] Burst_length : 0x0
[ 3708.300008] Burst_type : 0x1
[ 3708.303150] Beat_size : 0x2
[ 3708.306295] VQC : 0x0
[ 3708.309088] GRPSEC : 0x7e
[ 3708.311882] FALCONSEC : 0x0
[ 3708.315026] **************************************
[ 3708.319923] ------------[ cut here ]------------
[ 3708.319925] WARNING: CPU: 0 PID: 36096 at drivers/soc/tegra/cbb/tegra234-cbb.c:648 tegra234_cbb_isr+0x150/0x170
[ 3708.319933] Modules linked in: rfcomm xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype nft_compat nf_tables libcrc32c nvidia_drm(O) nvidia_modeset(O) nvidia_uvm(O) qrtr bridge stp llc usb_f_ncm usb_f_mass_storage nvidia(O) usb_f_acm governor_pod_scaling(O) u_serial usb_f_rndis u_ether libcomposite algif_hash algif_skcipher af_alg bnep snd_soc_tegra_controls(O) snd_soc_tegra_utils(O) snd_soc_tegra210_admaif snd_soc_tegra186_arad(O) snd_soc_tegra210_mixer snd_soc_tegra210_ope snd_soc_tegra_pcm snd_soc_tegra210_mvc snd_soc_tegra210_sfc snd_soc_tegra186_asrc snd_soc_tegra210_adx snd_soc_tegra210_amx snd_soc_tegra210_i2s snd_soc_tegra210_ahub nvvrs_pseq_rtc(O) tegra210_adma spidev rtk_btusb(O) btusb btrtl btintel btmtk btbcm bluetooth ecdh_generic ecc rtl8852ce(O) tegra_capture_coe(O) crct10dif_ce sm3_ce sm3 sha3_ce sha512_ce cfg80211 sha512_arm64 r8126(O) rfkill ina3221 ina238 tegra_aconnect tegra23x_psc(O) tegra_cactmon_mc_all(O) snd_hda_codec_hdmi at24 snd_hda_tegra

3

Hi @casey520_zhj,

From the logs, it seems there is an issue between the PCIe root port and the r8126 device. Specifically, it looks like the endpoint is not responding properly to PCIe transactions, which is consistent with the first errors observed:

[ 3702.062436] pcieport 0002:00:00.0: DPC: containment event, status:0x3f01 source:0x0000
[ 3702.062446] pcieport 0002:00:00.0: AER: Correctable error message received
[ 3702.062452] pcieport 0002:00:00.0: DPC: unmasked uncorrectable error detected
[ 3702.062496] pcieport 0002:00:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer
[ 3702.075394] pcieport 0002:00:00.0: [14] CmpltTO (First)
[ 3702.082046] r8126 0002:01:00.0: AER: can’t recover (no error_detected callback)

The CmpltTO (Completion Timeout) suggests that a PCIe transaction did not receive a response, which typically indicates that the endpoint (in this case, the r8126 NIC) stopped responding or the link became unstable.

Could you provide more details about your setup?

Since this type of issue can be hardware-related, it would be useful to check:

  • The physical connection between the PCIe interface and the r8126 device (to rule out poor contact or signal integrity issues)
  • Power stability for the PCIe device

From the software side, could you also check for additional dmesg logs related to PCIe and the r8126 driver? This may help identify if there are earlier warnings or related errors before the failure.

Best Regards
Carlos Quiros Gomez
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com
Website: www.ridgerun.com