Kernel panic when the pcie connecting with x86 and jetson orin nano

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

We insmod the tegra_vnet.ko in x86 board with pcie host and set the pcie function in Jetson Orin Nano.
And everthing is ok even the command ping is right util we run the iperf3 to stress the pcie communication.
It almost must happens and we get the below output after the iperf3 running a while.
Here is the command we run in two paltforms and the version of Jetpack is 5.1.3.
x86: iperf3 -s
Jetson: iperf3 -u -c 192.168.193.5 -b 2000M -t 999

[  989.300402] rcu: INFO: rcu_preempt self-detected stall on CPU
[  989.306334] rcu:     0-....: (5385 ticks this GP) idle=01a/1/0x4000000000000002 softirq=17903/17905 fqs=2093
[  989.316419] rcu: ====For debug only: Start Printing Blocked Tasks====<print_cpu_stall>
[  989.324747] rcu: ====For debug only: End Printing Blocked Tasks====<print_cpu_stall>
[ 1016.008401] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ksoftirqd/0:12]
[ 1016.015846] Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter br_netfilter lzo_rle lzo_compress zram overlay ramoops reed_solomon loop snd_soc_tegra186_asrc snd_soc_tegra186_arad snd_soc_tegra186_dspk snd_soc_tegra210_ope snd_soc_tegra210_iqc snd_soc_tegra210_mvc snd_soc_tegra210_afc snd_soc_tegra210_dmic snd_soc_tegra210_amx snd_soc_tegra210_adx snd_soc_tegra210_admaif snd_soc_tegra_pcm snd_soc_tegra210_i2s snd_soc_tegra210_mixer snd_soc_tegra210_sfc aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce snd_soc_tegra210_adsp sha256_arm64 sha1_ce snd_soc_tegra_machine_driver snd_soc_tegra_utils snd_soc_spdif_tx snd_soc_simple_card_utils snd_soc_tegra210_ahub fusb301 nvadsp snd_hda_codec_hdmi userspace_alert tegra_bpmp_thermal tegra210_adma snd_hda_tegra snd_hda_codec r8168 snd_hda_core nv_imx219 spi_tegra114 nvidia(O) binfmt_misc ina3221 pwm_fan nvgpu nvmap
[ 1016.015975]  ip_tables x_tables [last unloaded: mtd]
[ 1016.112906] CPU: 0 PID: 12 Comm: ksoftirqd/0 Tainted: G           O      5.10.192-tegra #1
[ 1016.121379] Hardware name: NVIDIA Orin Nano Developer Kit (DT)
[ 1016.127369] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=--)
[ 1016.133541] pc : skip_ftrace_call+0x2c/0x30
[ 1016.137830] lr : _dev_err+0x34/0x9c
[ 1016.141410] sp : ffff8000100bba90
[ 1016.144815] x29: ffff8000100bba90 x28: 0000000000007e10
[ 1016.150257] x27: ffff293a00c5e010 x26: 0000000000007e10
[ 1016.155709] x25: ffff293a0fcee100 x24: ffff800011191000
[ 1016.161160] x23: ffff293a11a2a800 x22: ffff293a00e85000
[ 1016.166603] x21: ffff293a11a2a800 x20: ffffb6b8eb6844b0
[ 1016.172051] x19: ffff293a00161d80 x18: ffff293a1a540c80
[ 1016.177503] x17: 0000000026c39d87 x16: ffffb6b8ea09e850
[ 1016.182961] x15: 0000000000000000 x14: ffffffffffffffff
[ 1016.188409] x13: 000000000000000e x12: 0000000000000010
[ 1016.193859] x11: 0000000000000270 x10: ffff293a168d5940
[ 1016.199311] x9 : 00000000000feba8 x8 : 00000000000001ff
[ 1016.204757] x7 : 0000000000000a20 x6 : ffff293a119e1d78
[ 1016.210202] x5 : 00000000feba8002 x4 : 00000000ff470040
[ 1016.215654] x3 : ffffb6b8ebc96000 x2 : ffffb6b8ea130390
[ 1016.221113] x1 : ffffb6b8eb6844b0 x0 : ffffb6b8ea130390
[ 1016.226562] Call trace:
[ 1016.229082]  skip_ftrace_call+0x2c/0x30
[ 1016.233017]  tvnet_ep_start_xmit+0x3ac/0x440
[ 1016.237402]  dev_hard_start_xmit+0x10c/0x330
[ 1016.241782]  sch_direct_xmit+0xf4/0x360
[ 1016.245716]  __qdisc_run+0x140/0x730
[ 1016.249382]  net_tx_action+0x14c/0x3c0
[ 1016.253225]  __do_softirq+0x140/0x3e8
[ 1016.256978]  run_ksoftirqd+0x50/0x60
[ 1016.260645]  smpboot_thread_fn+0x1c4/0x280
[ 1016.264843]  kthread+0x148/0x170
[ 1016.268158]  ret_from_fork+0x10/0x18
[ 1016.271821] watchdog: ====For debug only: Start Printing Blocked Tasks====<watchdog_timer_fn>
[ 1016.280641] watchdog: ====For debug only: End Printing Blocked Tasks====<watchdog_timer_fn>
[ 1016.289197] Kernel panic - not syncing: softlockup: hung tasks
[ 1016.295173] CPU: 0 PID: 12 Comm: ksoftirqd/0 Tainted: G           O L    5.10.192-tegra #1
[ 1016.303645] Hardware name: NVIDIA Orin Nano Developer Kit (DT)
[ 1016.309628] Call trace:
[ 1016.312133]  dump_backtrace+0x0/0x1e0
[ 1016.315888]  show_stack+0x30/0x40
[ 1016.319294]  dump_stack+0xf0/0x130
[ 1016.322777]  panic+0x1a4/0x38c
[ 1016.325913]  watchdog_timer_fn+0x270/0x2d0
[ 1016.330114]  __hrtimer_run_queues+0x148/0x380
[ 1016.334578]  hrtimer_interrupt+0xf0/0x250
[ 1016.338690]  arch_timer_handler_phys+0x40/0x50
[ 1016.343246]  handle_percpu_devid_irq+0x90/0x280
[ 1016.347891]  generic_handle_irq+0x40/0x60
[ 1016.352003]  __handle_domain_irq+0x70/0xd0
[ 1016.356202]  gic_handle_irq+0x68/0x134
[ 1016.360046]  el1_irq+0xd0/0x180
[ 1016.363262]  skip_ftrace_call+0x2c/0x30
[ 1016.367204]  tvnet_ep_start_xmit+0x3ac/0x440
[ 1016.371587]  dev_hard_start_xmit+0x10c/0x330
[ 1016.375964]  sch_direct_xmit+0xf4/0x360
[ 1016.379894]  __qdisc_run+0x140/0x730
[ 1016.383558]  net_tx_action+0x14c/0x3c0
[ 1016.387407]  __do_softirq+0x140/0x3e8
[ 1016.391160]  run_ksoftirqd+0x50/0x60
[ 1016.394827]  smpboot_thread_fn+0x1c4/0x280
[ 1016.399023]  kthread+0x148/0x170
[ 1016.402338]  ret_from_fork+0x10/0x18
[ 1016.406013] SMP: stopping secondary CPUs
[ 1016.410040] Kernel Offset: 0x36b8d9f70000 from 0xffff800010000000
[ 1016.416289] PHYS_OFFSET: 0xffffd6c700000000
[ 1016.420583] CPU features: 0x08040006,4a80aa38
[ 1016.425054] Memory Limit: none
[ 1016.435032] ---[ end Kernel panic - not syncing: softlockup: hung tasks ]---
2

And we exchange the direction of iperfs, we got annother result. The iperf3 ouput is in below. It seems no data is transmitting in pcie.
jetson:iperf3 -s

[  5] 157.00-158.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 158.00-159.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 159.00-160.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 160.00-161.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 161.00-162.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 162.00-163.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 163.00-164.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 164.00-165.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 165.00-166.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 166.00-167.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 167.00-168.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 168.00-169.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 169.00-170.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 170.00-171.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 171.00-172.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 172.00-173.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 173.00-174.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 174.00-175.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 175.00-176.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 176.00-177.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 177.00-178.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 178.00-179.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 179.00-180.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 180.00-181.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)
[  5] 181.00-182.00 sec  0.00 Bytes  0.00 bits/sec  0.076 ms  0/0 (0%)

x86: iperf3 -u -c 192.168.193.6 -b 4000M -t 9999

[  5] 205.00-206.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 206.00-207.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 207.00-208.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 208.00-209.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 209.00-210.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 210.00-211.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 211.00-212.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 212.00-213.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 213.00-214.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 214.00-215.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 215.00-216.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 216.00-217.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 217.00-218.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 218.00-219.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 219.00-220.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 220.00-221.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 221.00-222.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 222.00-223.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 223.00-224.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 224.00-225.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 225.00-226.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 226.00-227.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 227.00-228.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 228.00-229.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 229.00-230.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 230.00-231.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 231.00-232.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 232.00-233.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 233.00-234.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 234.00-235.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 235.00-236.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 236.00-237.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 237.00-238.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 238.00-239.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 239.00-240.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 240.00-241.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 241.00-242.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 242.00-243.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 243.00-244.00 sec  0.00 Bytes  0.00 bits/sec  0
[  5] 244.00-245.00 sec  0.00 Bytes  0.00 bits/sec  0
4

The driver we provided only supports running on Jetson to Jetson.

5

So is there the official driver for x86?
The tegra_vnet driver is found in the forums.

6

Currently there is no support for it.

7

Are there some example for memory mapping communication?

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.