Measuring Jetson TX2 memory controller throughput

I have a Jetson TX2i connected to a Jetson TX2 via a PCIe switch. Each processor is connected to an NTB port on the PCIe switch.

The PCIe connection is gen 2, x4 with a theoretical max rate of 2GBytes/sec. To test throughput between the to processors, I am using the ntb_perf kernel driver.

When I push data from the TX2 to the TX2i, I get an overall data rate of 282MB/sec. This is well below the theoretical max. One of the CPU cores is pegged at 100% system. Thinking the limitation was CPU performance, I added a thread to the test allowing 2 CPU cores to move data. This resulted in each thread moving 141MB/sec so the aggregate bandwidth remained 282MB/sec.

When I push data from the TX2i to the TX2, I get an overall data rate of 410MB/sec. Although not close to theoretical, this is much better than the other direction.

So, my questions are the following.

  1. Why is the data transfer from TX2i to TX2 significantly better than the other direction? Is the TX2i DDR or memory controller capable of performing at a higher bandwidth?

  2. I don’t believe PCIe or CPU performance are limiting my transfer? So, what is?

  3. Is there a way to measure memory controller throughput?

Hi Jeremy,

Could you also try TX2 to TX2 transfer and TX2i to TX2i transfer?

Also, what release are you using?

I used Jetpack 4.2.1 with L4T R32.2.1. The kernel version is 4.9.140-tegra.

I will see what hardware is available to try the TX2->TX2 and TX2i->TX2i. I’ll post my results.

I have data for a TX2 to TX2 setup. The summary points are first. Details follow.

  1. Ran ‘nvpmodel -m 0’ and ‘jetson_clocks’ before all tests

  2. ntb_perf test resulted in 282MB/sec in both directions. CPU2 was pegged at 100% (system). This is different from TX2i to TX2 test where I got 410MB/sec.

  3. Dual thread ntb_perf test resulted in 141MB/sec transfered by each thread. No throughput improvement over single threaded test. CPU1 and CPU2 were pegged at 100% (system).

  4. iperf test using ntb_transport/ntb_netdev resulted in 800Mbits/sec. This is less than the performance I got with TX2i and TX2.

Below are the details.

NTB_PERF TESTS
On both machines
sudo nvpmodel -m 0
sudo jetson_clocks

[b]----------------------------------------------------------

  • 1 thread test (tx2-dev to tx2-dev2)
    ----------------------------------------------------------[/b]
    root@tx2-dev:/home/nvidia/pallotta# echo 1 > /sys/kernel/debug/ntb_perf/0000:01:00.1/run

CPU2 is pegged (system = 100%)

[416754.998816] kthread ntb_perf 0 starting…
[416785.439730] ntb_perf 0: copied 8589934592 bytes
[416785.439738] ntb_perf 0: lasted 30440733 usecs
[416785.439741] ntb_perf 0: MBytes/s: 282

[b]----------------------------------------------------------

  • 2 thread test (tx2-dev to tx2-dev2)
    ----------------------------------------------------------[/b]
    root@tx2-dev:/home/nvidia/pallotta# echo 2 > /sys/kernel/debug/ntb_perf/0000:01:00.1/threads
    root@tx2-dev:/home/nvidia/pallotta# echo 1 > /sys/kernel/debug/ntb_perf/0000:01:00.1/run

CPU1 and CPU2 are pegged (system = 100%)

[416872.721687] kthread ntb_perf 0 starting…
[416872.721759] kthread ntb_perf 1 starting…
[416933.146275] ntb_perf 1: copied 8589934592 bytes
[416933.146283] ntb_perf 1: lasted 60424275 usecs
[416933.146287] ntb_perf 1: MBytes/s: 142
[416933.549142] ntb_perf 0: copied 8589934592 bytes
[416933.549150] ntb_perf 0: lasted 60827144 usecs
[416933.549154] ntb_perf 0: MBytes/s: 141

[b]----------------------------------------------------------

  • 1 thread test (tx2-dev2 to tx2-dev)
    ----------------------------------------------------------[/b]
    root@tx2-dev2:/home/nvidia/pallotta# echo 1 > /sys/kernel/debug/ntb_perf/0000:01:00.1/run

CPU1 is pegged (system = 100%)

[ 1013.740781] Fix seg_order to 20
[ 1013.740910] kthread ntb_perf 0 starting…
[ 1044.191753] ntb_perf 0: copied 8589934592 bytes
[ 1044.191766] ntb_perf 0: lasted 30450649 usecs
[ 1044.191773] ntb_perf 0: MBytes/s: 282

[b]----------------------------------------------------------

  • 2 thread test (tx2-dev2 to tx2-dev)
    ----------------------------------------------------------[/b]
    root@tx2-dev2:/home/nvidia/pallotta# echo 2 > /sys/kernel/debug/ntb_perf/0000:01:00.1/threads
    root@tx2-dev2:/home/nvidia/pallotta# echo 1 > /sys/kernel/debug/ntb_perf/0000:01:00.1/run

CPU1 and CPU2 are pegged (system = 100%)

[ 1120.627455] kthread ntb_perf 0 starting…
[ 1120.627517] kthread ntb_perf 1 starting…
[ 1179.232134] ntb_perf 1: copied 8589934592 bytes
[ 1179.232192] ntb_perf 1: lasted 58604122 usecs
[ 1179.233829] ntb_perf 1: MBytes/s: 146
[ 1181.463946] ntb_perf 0: copied 8589934592 bytes
[ 1181.463953] ntb_perf 0: lasted 60836176 usecs
[ 1181.463958] ntb_perf 0: MBytes/s: 141

tx2-dev (client) to tx2-dev2 (server) = ~800MB/sec

Iperf Server command line
root@tx2-dev2:/home/nvidia/pallotta# iperf3 -s -A 0 -i 1 -V

Iperf Client command line
root@tx2-dev:/home/nvidia/pallotta# iperf3 -c 192.168.100.3 -P 2 -A 0 -M -t 30 -w 16M -l 1M -b 0
Connecting to host 192.168.100.3, port 5201
[ 4] local 192.168.100.2 port 36792 connected to 192.168.100.3 port 5201
[ 6] local 192.168.100.2 port 36794 connected to 192.168.100.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.02 sec 98.5 MBytes 807 Mbits/sec 0 639 KBytes
[ 6] 0.00-1.02 sec 98.1 MBytes 804 Mbits/sec 0 639 KBytes
[SUM] 0.00-1.02 sec 197 MBytes 1.61 Gbits/sec 0


[ 4] 1.02-2.02 sec 95.4 MBytes 806 Mbits/sec 0 639 KBytes
[ 6] 1.02-2.02 sec 94.9 MBytes 801 Mbits/sec 0 639 KBytes
[SUM] 1.02-2.02 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 2.02-3.01 sec 95.0 MBytes 804 Mbits/sec 0 639 KBytes
[ 6] 2.02-3.01 sec 94.9 MBytes 803 Mbits/sec 0 639 KBytes
[SUM] 2.02-3.01 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 3.01-4.00 sec 95.0 MBytes 804 Mbits/sec 0 639 KBytes
[ 6] 3.01-4.00 sec 94.9 MBytes 803 Mbits/sec 0 639 KBytes
[SUM] 3.01-4.00 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 4.00-5.02 sec 97.5 MBytes 804 Mbits/sec 0 639 KBytes
[ 6] 4.00-5.02 sec 97.4 MBytes 803 Mbits/sec 0 639 KBytes
[SUM] 4.00-5.02 sec 195 MBytes 1.61 Gbits/sec 0


[ 4] 5.02-6.01 sec 95.1 MBytes 805 Mbits/sec 0 639 KBytes
[ 6] 5.02-6.01 sec 94.9 MBytes 803 Mbits/sec 0 639 KBytes
[SUM] 5.02-6.01 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 6.01-7.00 sec 95.3 MBytes 805 Mbits/sec 0 639 KBytes
[ 6] 6.01-7.00 sec 94.9 MBytes 802 Mbits/sec 0 639 KBytes
[SUM] 6.01-7.00 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 7.00-8.02 sec 98.0 MBytes 807 Mbits/sec 0 639 KBytes
[ 6] 7.00-8.02 sec 97.4 MBytes 802 Mbits/sec 0 639 KBytes
[SUM] 7.00-8.02 sec 195 MBytes 1.61 Gbits/sec 0


[ 4] 8.02-9.01 sec 95.1 MBytes 805 Mbits/sec 0 639 KBytes
[ 6] 8.02-9.01 sec 94.9 MBytes 803 Mbits/sec 0 639 KBytes
[SUM] 8.02-9.01 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 9.01-10.01 sec 95.5 MBytes 804 Mbits/sec 0 639 KBytes
[ 6] 9.01-10.01 sec 94.9 MBytes 799 Mbits/sec 0 639 KBytes
[SUM] 9.01-10.01 sec 190 MBytes 1.60 Gbits/sec 0


[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.01 sec 960 MBytes 805 Mbits/sec 0 sender
[ 4] 0.00-10.01 sec 960 MBytes 805 Mbits/sec receiver
[ 6] 0.00-10.01 sec 957 MBytes 802 Mbits/sec 0 sender
[ 6] 0.00-10.01 sec 957 MBytes 802 Mbits/sec receiver
[SUM] 0.00-10.01 sec 1.87 GBytes 1.61 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 1.87 GBytes 1.61 Gbits/sec receiver

iperf Done.

tx2-dev2 (client) to tx2-dev (server) = ~800MB/sec

Iperf Server command line
root@tx2-dev:/home/nvidia/pallotta# iperf3 -s -A 0 -i 1 -V

Iperf Client command line
root@tx2-dev2:/home/nvidia/pallotta# iperf3 -c 192.168.100.2 -P 2 -A 0 -M -t 30 -w 16M -l 1M -b 0
Connecting to host 192.168.100.2, port 5201
[ 4] local 192.168.100.3 port 39746 connected to 192.168.100.2 port 5201
[ 6] local 192.168.100.3 port 39748 connected to 192.168.100.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.02 sec 98.5 MBytes 808 Mbits/sec 0 639 KBytes
[ 6] 0.00-1.02 sec 98.3 MBytes 806 Mbits/sec 0 639 KBytes
[SUM] 0.00-1.02 sec 197 MBytes 1.61 Gbits/sec 0


[ 4] 1.02-2.02 sec 94.9 MBytes 802 Mbits/sec 0 639 KBytes
[ 6] 1.02-2.02 sec 95.8 MBytes 809 Mbits/sec 0 639 KBytes
[SUM] 1.02-2.02 sec 191 MBytes 1.61 Gbits/sec 0


[ 4] 2.02-3.01 sec 95.5 MBytes 802 Mbits/sec 0 639 KBytes
[ 6] 2.02-3.01 sec 96.1 MBytes 808 Mbits/sec 0 639 KBytes
[SUM] 2.02-3.01 sec 192 MBytes 1.61 Gbits/sec 0


[ 4] 3.01-4.00 sec 94.9 MBytes 805 Mbits/sec 0 639 KBytes
[ 6] 3.01-4.00 sec 94.9 MBytes 805 Mbits/sec 0 639 KBytes
[SUM] 3.01-4.00 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 4.00-5.02 sec 97.4 MBytes 806 Mbits/sec 0 639 KBytes
[ 6] 4.00-5.02 sec 97.4 MBytes 806 Mbits/sec 0 639 KBytes
[SUM] 4.00-5.02 sec 195 MBytes 1.61 Gbits/sec 0


[ 4] 5.02-6.01 sec 95.0 MBytes 806 Mbits/sec 0 639 KBytes
[ 6] 5.02-6.01 sec 94.9 MBytes 805 Mbits/sec 0 639 KBytes
[SUM] 5.02-6.01 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 6.01-7.02 sec 97.4 MBytes 806 Mbits/sec 0 639 KBytes
[ 6] 6.01-7.02 sec 97.4 MBytes 806 Mbits/sec 0 639 KBytes
[SUM] 6.01-7.02 sec 195 MBytes 1.61 Gbits/sec 0


[ 4] 7.02-8.01 sec 94.9 MBytes 805 Mbits/sec 0 639 KBytes
[ 6] 7.02-8.01 sec 94.9 MBytes 805 Mbits/sec 0 639 KBytes
[SUM] 7.02-8.01 sec 190 MBytes 1.61 Gbits/sec 0


[ 4] 8.01-9.02 sec 97.4 MBytes 804 Mbits/sec 0 639 KBytes
[ 6] 8.01-9.02 sec 97.4 MBytes 804 Mbits/sec 0 639 KBytes
[SUM] 8.01-9.02 sec 195 MBytes 1.61 Gbits/sec 0


[ 4] 9.02-10.01 sec 94.9 MBytes 805 Mbits/sec 0 639 KBytes
[ 6] 9.02-10.01 sec 94.9 MBytes 805 Mbits/sec 0 639 KBytes
[SUM] 9.02-10.01 sec 190 MBytes 1.61 Gbits/sec 0


[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.01 sec 961 MBytes 805 Mbits/sec 0 sender
[ 4] 0.00-10.01 sec 961 MBytes 805 Mbits/sec receiver
[ 6] 0.00-10.01 sec 962 MBytes 806 Mbits/sec 0 sender
[ 6] 0.00-10.01 sec 962 MBytes 806 Mbits/sec receiver
[SUM] 0.00-10.01 sec 1.88 GBytes 1.61 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 1.88 GBytes 1.61 Gbits/sec receiver

iperf Done.

I have data for a TX2i to TX2i setup. The summary points are first. Details follow.

  1. Ran ‘nvpmodel -m 0’ and ‘jetson_clocks’ before all tests

  2. ntb_perf test resulted in 410MB/sec in both directions. CPU1 was pegged at 100% (system).

  3. Dual thread ntb_perf test resulted in ~205MB/sec transferred by each thread. No throughput improvement over single threaded test. CPU1 and CPU2 were pegged at 100% (system).

  4. iperf test using ntb_transport/ntb_netdev resulted in 1Gbits/sec. This is slightly better than the performance I got with TX2i and TX2.

Below are the details.

NTB_PERF TESTS
On both machines
sudo nvpmodel -m 0
sudo jetson_clocks

[b]----------------------------------------------------------

  • 1 thread test (tx2i-dev to tx2i-dev2)
    ----------------------------------------------------------[/b]

CPU1 is pegged (system = 100%)

[78808.094070] Fix seg_order to 20
[79073.205917] kthread ntb_perf 0 starting…
[79094.134978] ntb_perf 0: copied 8589934592 bytes
[79094.134986] ntb_perf 0: lasted 20928868 usecs
[79094.134992] ntb_perf 0: MBytes/s: 410

[b]----------------------------------------------------------

  • 2 thread test (tx2i-dev to tx2i-dev2)
    ----------------------------------------------------------[/b]
    root@tx2i-dev:/home/nvidia/pallotta# echo 2 > /sys/kernel/debug/ntb_perf/0000:01:00.1/threads
    root@tx2i-dev:/home/nvidia/pallotta# echo 1 > /sys/kernel/debug/ntb_perf/0000:01:00.1/run

CPU1 and CPU2 are pegged (system = 100%)

[79217.962330] kthread ntb_perf 0 starting…
[79217.962397] kthread ntb_perf 1 starting…
[79259.081433] ntb_perf 0: copied 8589934592 bytes
[79259.081480] ntb_perf 0: lasted 41113110 usecs
[79259.081506] ntb_perf 0: MBytes/s: 208
[79259.800573] ntb_perf 1: copied 8589934592 bytes
[79259.800581] ntb_perf 1: lasted 41832344 usecs
[79259.800586] ntb_perf 1: MBytes/s: 205

[b]----------------------------------------------------------

  • 1 thread test (tx2i-dev2 to tx2i-dev)
    ----------------------------------------------------------[/b]

CPU1 is pegged (system = 100%)

root@tx2i-dev2:/home/nvidia/pallotta# echo 1 > /sys/kernel/debug/ntb_perf/0000:01:00.1/run
[ 1605.347186] Fix seg_order to 20
[ 1605.347320] kthread ntb_perf 0 starting…
[ 1626.289288] ntb_perf 0: copied 8589934592 bytes
[ 1626.289299] ntb_perf 0: lasted 20941703 usecs
[ 1626.289307] ntb_perf 0: MBytes/s: 410

[b]----------------------------------------------------------

  • 2 thread test (tx2i-dev2 to tx2i-dev)
    ----------------------------------------------------------[/b]
    root@tx2i-dev2:/home/nvidia/pallotta# echo 2 > /sys/kernel/debug/ntb_perf/0000:01:00.1/threads
    root@tx2i-dev2:/home/nvidia/pallotta# echo 1 > /sys/kernel/debug/ntb_perf/0000:01:00.1/run

CPU1 and CPU2 are pegged (system = 100%)

[ 1756.741324] kthread ntb_perf 0 starting…
[ 1756.741392] kthread ntb_perf 1 starting…
[ 1796.833239] ntb_perf 0: copied 8589934592 bytes
[ 1796.833293] ntb_perf 0: lasted 40091388 usecs
[ 1796.833314] ntb_perf 0: MBytes/s: 214
[ 1798.614288] ntb_perf 1: copied 8589934592 bytes
[ 1798.614298] ntb_perf 1: lasted 41872645 usecs
[ 1798.614305] ntb_perf 1: MBytes/s: 205

tx2i-dev (client) to tx2i-dev2 (server) = ~1Gbit/sec

Iperf Server command line
root@tx2i-dev2:/home/nvidia/pallotta# iperf3 -s -A 0 -i 1 -V

Iperf Client command line
root@tx2i-dev:/home/nvidia/pallotta# iperf3 -c 192.168.100.9 -P 2 -A 0 -M -t 30 -w 16M -l 1M -b 0
Connecting to host 192.168.100.9, port 5201
[ 4] local 192.168.100.3 port 46292 connected to 192.168.100.9 port 5201
[ 6] local 192.168.100.3 port 46294 connected to 192.168.100.9 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.01 sec 123 MBytes 1.02 Gbits/sec 0 639 KBytes
[ 6] 0.00-1.01 sec 123 MBytes 1.02 Gbits/sec 0 639 KBytes
[SUM] 0.00-1.01 sec 245 MBytes 2.03 Gbits/sec 0


[ 4] 1.01-2.00 sec 120 MBytes 1.02 Gbits/sec 0 639 KBytes
[ 6] 1.01-2.00 sec 120 MBytes 1.02 Gbits/sec 0 639 KBytes
[SUM] 1.01-2.00 sec 240 MBytes 2.03 Gbits/sec 0


[ 4] 2.00-3.01 sec 122 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 2.00-3.01 sec 122 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 2.00-3.01 sec 245 MBytes 2.03 Gbits/sec 0


[ 4] 3.01-4.00 sec 120 MBytes 1.02 Gbits/sec 0 639 KBytes
[ 6] 3.01-4.00 sec 120 MBytes 1.02 Gbits/sec 0 639 KBytes
[SUM] 3.01-4.00 sec 239 MBytes 2.03 Gbits/sec 0


[ 4] 4.00-5.00 sec 121 MBytes 1.02 Gbits/sec 0 639 KBytes
[ 6] 4.00-5.00 sec 121 MBytes 1.02 Gbits/sec 0 639 KBytes
[SUM] 4.00-5.00 sec 242 MBytes 2.03 Gbits/sec 0


[ 4] 5.00-6.02 sec 123 MBytes 1.02 Gbits/sec 0 639 KBytes
[ 6] 5.00-6.02 sec 123 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 5.00-6.02 sec 246 MBytes 2.03 Gbits/sec 0


[ 4] 6.02-7.01 sec 120 MBytes 1.02 Gbits/sec 0 639 KBytes
[ 6] 6.02-7.01 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 6.02-7.01 sec 239 MBytes 2.03 Gbits/sec 0


[ 4] 7.01-8.00 sec 121 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 7.01-8.00 sec 121 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 7.01-8.00 sec 241 MBytes 2.03 Gbits/sec 0


[ 4] 8.00-9.01 sec 121 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 8.00-9.01 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 8.00-9.01 sec 241 MBytes 2.02 Gbits/sec 0


[ 4] 9.01-10.01 sec 122 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 9.01-10.01 sec 121 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 9.01-10.01 sec 243 MBytes 2.03 Gbits/sec 0


[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.01 sec 1.18 GBytes 1.02 Gbits/sec 0 sender
[ 4] 0.00-10.01 sec 1.18 GBytes 1.02 Gbits/sec receiver
[ 6] 0.00-10.01 sec 1.18 GBytes 1.01 Gbits/sec 0 sender
[ 6] 0.00-10.01 sec 1.18 GBytes 1.01 Gbits/sec receiver
[SUM] 0.00-10.01 sec 2.37 GBytes 2.03 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 2.36 GBytes 2.03 Gbits/sec receiver

iperf Done.

tx2i-dev2 (client) to tx2i-dev (server) = ~1Gbit/sec

Iperf Server command line
root@tx2i-dev:/home/nvidia/pallotta# iperf3 -s -A 0 -i 1 -V

Iperf Client command line
root@tx2i-dev2:/home/nvidia/pallotta# iperf3 -c 192.168.100.3 -P 2 -A 0 -M -t 30 -w 16M -l 1M -b 0
Connecting to host 192.168.100.3, port 5201
[ 4] local 192.168.100.9 port 52272 connected to 192.168.100.3 port 5201
[ 6] local 192.168.100.9 port 52274 connected to 192.168.100.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 0.00-1.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 0.00-1.00 sec 240 MBytes 2.01 Gbits/sec 0


[ 4] 1.00-2.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 1.00-2.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 1.00-2.00 sec 240 MBytes 2.01 Gbits/sec 0


[ 4] 2.00-3.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 2.00-3.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 2.00-3.00 sec 240 MBytes 2.01 Gbits/sec 0


[ 4] 3.00-4.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 3.00-4.00 sec 120 MBytes 1.00 Gbits/sec 0 639 KBytes
[SUM] 3.00-4.00 sec 240 MBytes 2.01 Gbits/sec 0


[ 4] 4.00-5.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 4.00-5.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 4.00-5.00 sec 240 MBytes 2.01 Gbits/sec 0


[ 4] 5.00-6.01 sec 121 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 5.00-6.01 sec 121 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 5.00-6.01 sec 241 MBytes 2.01 Gbits/sec 0


[ 4] 6.01-7.01 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 6.01-7.01 sec 121 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 6.01-7.01 sec 241 MBytes 2.02 Gbits/sec 0


[ 4] 7.01-8.00 sec 119 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 7.01-8.00 sec 119 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 7.01-8.00 sec 238 MBytes 2.02 Gbits/sec 0


[ 4] 8.00-9.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 8.00-9.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 8.00-9.00 sec 240 MBytes 2.02 Gbits/sec 0


[ 4] 9.00-10.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[ 6] 9.00-10.00 sec 120 MBytes 1.01 Gbits/sec 0 639 KBytes
[SUM] 9.00-10.00 sec 240 MBytes 2.02 Gbits/sec 0


[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec receiver
[ 6] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec 0 sender
[ 6] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec receiver
[SUM] 0.00-10.00 sec 2.34 GBytes 2.01 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 2.34 GBytes 2.01 Gbits/sec receiver

iperf Done.

Could you share what pcie switch you are using?

Hi,

Can you share the boot log for both TX2i and TX2?

Also, can you share the dump of the below node:
root@tegra-ubuntu:/home/ubuntu# cat /sys/kernel/debug/bpmp/debug/emc/dram_info

In general, TX2i’s memory BW is less than TX2 since DRAM ECC is enabled in TX2i

regards
Bibek

Our current tests are being done with an eval card containing a Microsemi Gen3 Switchtec PM8536. Our current plans are to utilize a Microsemi Gen3 Switchtec PM8571B in the final system. We are connecting the Jetson TX2/TX2i’s to x4 ports on the eval card. The lanes are negotiating to gen2 x4, as expected. Below is the lspci dump for the non-transparent bridge endpoint on one of the TX2i.

nvidia@tx2i-dev:~$ sudo lspci -s 1:0.1 -vv
01:00.1 Bridge: PMC-Sierra Inc. Device 8536
Subsystem: PMC-Sierra Inc. Device 8536
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort+ <TAbort- SERR- <PERR- INTx-
Region 0: Memory at 4c000000 (64-bit, prefetchable) [disabled]
Region 2: Memory at 48000000 (64-bit, prefetchable) [disabled]
Capabilities: [40] MSI: Enable- Count=1/4 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [50] MSI-X: Enable- Count=4 Masked-
Vector table: BAR=0 offset=0007a000
PBA: BAR=0 offset=0007b000
Capabilities: [5c] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138 v1] #12
Capabilities: [168 v1] Device Serial Number 50-0e-00-4a-00-00-00-01
Capabilities: [7f8 v1] Vendor Specific Information: ID=ffff Rev=1 Len=808 <?>
Kernel driver in use: switchtec
Kernel modules: switchtec

I assume by “boot log”, you want the output of “dmesg”. If not, let me know what you would like.

For first TX2i, the output is the following. The second TX2i will follow the first.

root@tx2i-dev:/home/nvidia# cat /sys/kernel/debug/bpmp/debug/emc/dram_info
Manufacturer ID (MR5): 6
Revision ID-1 (MR6): 4
Revision ID-2 (MR7): 0
IO Width/Density/Type (MR8): 0x08

root@tx2i-dev:/home/nvidia# dmesg
[ 0.000000] Booting Linux on physical CPU 0x100
[ 0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-770) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Mon Aug 12 21:29:52 PDT 2019
[ 0.000000] Boot CPU: AArch64 Processor [411fd073]
[ 0.000000] OF: fdt:memory scan node memory@80000000, reg size 16416,
[ 0.000000] OF: fdt: - 80000000 , 70000000
[ 0.000000] OF: fdt: - f0200000 , 145600000
[ 0.000000] OF: fdt: - 235e00000 , 200000
[ 0.000000] OF: fdt: - 236600000 , 200000
[ 0.000000] OF: fdt: - 237000000 , 200000
[ 0.000000] earlycon: uart8250 at MMIO32 0x0000000003100000 (options ‘’)
[ 0.000000] bootconsole [uart8250] enabled
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb0_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb0_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb1_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb1_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb2_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb2_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node ramoops_carveout, compatible id nvidia,ramoops
[ 0.000000] cma: Reserved 64 MiB at 0x00000000fc000000
[ 0.000000] On node 0 totalpages: 1793024
[ 0.000000] DMA zone: 8192 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 523776 pages, LIFO batch:31
[ 0.000000] Normal zone: 19912 pages used for memmap
[ 0.000000] Normal zone: 1269248 pages, LIFO batch:31
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.0 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.1
[ 0.000000] percpu: Embedded 25 pages/cpu @ffffffc1b676a000 s61592 r8192 d32616 u102400
[ 0.000000] pcpu-alloc: s61592 r8192 d32616 u102400 alloc=25*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
[ 0.000000] Speculative Store Bypass Disable mitigation not required
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1764920
[ 0.000000] Kernel command line: root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x3100000 nvdumper_reserved=0x2372e0000 gpt usbcore.old_scheme_first=1 tegraid=18.1.2.0.0 maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.2031647.1 bl_prof_dataptr=0x10000@0x235840000 sdhci_tegra.en_boot_part_access=1 quiet
[ 0.000000] log_buf_len individual max cpu contribution: 32768 bytes
[ 0.000000] log_buf_len total cpu_extra contributions: 163840 bytes
[ 0.000000] log_buf_len min size: 32768 bytes
[ 0.000000] log_buf_len: 262144 bytes
[ 0.000000] early log buf free: 29424(89%)
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[ 0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[ 0.000000] Memory: 6937008K/7172096K available (15230K kernel code, 2914K rwdata, 6628K rodata, 8576K init, 609K bss, 169552K reserved, 65536K cma-reserved)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] modules : 0xffffff8000000000 - 0xffffff8008000000 ( 128 MB)
[ 0.000000] vmalloc : 0xffffff8008000000 - 0xffffffbebfff0000 ( 250 GB)
[ 0.000000] .text : 0xffffff8008080000 - 0xffffff8008f60000 ( 15232 KB)
[ 0.000000] .rodata : 0xffffff8008f60000 - 0xffffff80095e0000 ( 6656 KB)
[ 0.000000] .init : 0xffffff80095e0000 - 0xffffff8009e40000 ( 8576 KB)
[ 0.000000] .data : 0xffffff8009e40000 - 0xffffff800a118808 ( 2915 KB)
[ 0.000000] .bss : 0xffffff800a118808 - 0xffffff800a1b0dbc ( 610 KB)
[ 0.000000] fixed : 0xffffffbefe7fd000 - 0xffffffbefec00000 ( 4108 KB)
[ 0.000000] PCI I/O : 0xffffffbefee00000 - 0xffffffbeffe00000 ( 16 MB)
[ 0.000000] vmemmap : 0xffffffbf00000000 - 0xffffffc000000000 ( 4 GB maximum)
[ 0.000000] 0xffffffbf00000000 - 0xffffffbf06dc8000 ( 109 MB actual)
[ 0.000000] memory : 0xffffffc000000000 - 0xffffffc1b7200000 ( 7026 MB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 64.
[ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=6.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=6
[ 0.000000] NR_IRQS:64 nr_irqs:64 0
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] arm_arch_timer: Architected cp15 timer(s) running at 31.25MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xe6a171046, max_idle_ns: 881590405314 ns
[ 0.000003] sched_clock: 56 bits at 31MHz, resolution 32ns, wraps every 4398046511088ns
[ 0.001581] Console: colour dummy device 80x25
[ 0.001586] console [tty0] enabled
[ 0.001590] bootconsole [uart8250] disabled
[ 0.001597] kmemleak: Kernel memory leak detector disabled
[ 0.001607] Calibrating delay loop (skipped), value calculated using timer frequency… 62.50 BogoMIPS (lpj=125000)
[ 0.001615] pid_max: default: 32768 minimum: 301
[ 0.001932] Security Framework initialized
[ 0.002091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes)
[ 0.002095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes)
[ 0.002705] ftrace: allocating 46938 entries in 184 pages
[ 0.128981] sched-energy: Sched-energy-costs installed from DT
[ 0.128993] ASID allocator initialised with 65536 entries
[ 0.167977] tegra-id: chipid=21817.
[ 0.167983] tegra-id: opt_subrevision=1.
[ 0.167997] Tegra Revision: A02p SKU: 0x90 CPU Process: 0 SoC Process: 0
[ 0.168005] DTS File Name: /dvs/git/dirty/git-master_linux/kernel/kernel-4.9/arch/arm64/boot/dts/…/…/…/…/…/…/hardware/nvidia/platform/t18x/quill/kernel-dts/tegra186-quill-p3489-1000-a00-00-base.dts
[ 0.168010] DTB Build time:
[ 0.213728] CPU1: Booted secondary processor [4e0f0030]
[ 0.245332] CPU2: Booted secondary processor [4e0f0030]
[ 0.277124] CPU3: Booted secondary processor [411fd073]
[ 0.309116] CPU4: Booted secondary processor [411fd073]
[ 0.341105] CPU5: Booted secondary processor [411fd073]
[ 0.341162] Brought up 6 CPUs
[ 0.341166] SMP: Total of 6 processors activated.
[ 0.341171] CPU features: detected feature: 32-bit EL0 Support
[ 0.341934] CPU: All CPU(s) started at EL2
[ 0.341986] alternatives: patching kernel code
[ 0.351503] devtmpfs: initialized
[ 0.388442] Initilizing CustomIPI irq domain
[ 0.388632] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.388644] futex hash table entries: 2048 (order: 5, 131072 bytes)
[ 0.389572] pinctrl core: initialized pinctrl subsystem
[ 0.389795] OS set in device tree is not L4T.
[ 0.389961] regulator-dummy: no parameters
[ 0.390119] Initializing plugin-manager
[ 0.390175] Plugin module not found
[ 0.390212] Plugin-manager status disabled
[ 0.392076] NET: Registered protocol family 16
[ 0.392793] pstore: using zlib compression
[ 0.392810] console [pstore-1] enabled
[ 0.392813] pstore: Registered ramoops as persistent store backend
[ 0.392816] ramoops: attached 0x200000@0x235880000, ecc: 0/0
[ 0.408683] cpuidle: using governor menu
[ 0.410223] bpmp: waiting for handshake
[ 0.410226] bpmp: handshake completed
[ 0.410237] bpmp: synchronizing channels
[ 0.410255] bpmp: channels synchronized
[ 0.410258] bpmp: mail init ok
[ 0.412336] arm-smmu 12000000.iommu: probing hardware configuration…
[ 0.412340] arm-smmu 12000000.iommu: SMMUv2 with:
[ 0.412344] arm-smmu 12000000.iommu: stage 1 translation
[ 0.412347] arm-smmu 12000000.iommu: stage 2 translation
[ 0.412349] arm-smmu 12000000.iommu: nested translation
[ 0.412355] arm-smmu 12000000.iommu: stream matching with 128 register groups, mask 0x7f80
[ 0.412360] arm-smmu 12000000.iommu: SMMU address space size (0x800000) differs from mapped region size (0x1000000)!
[ 0.412364] arm-smmu 12000000.iommu: 64 context banks (0 stage-2 only)
[ 0.412368] arm-smmu 12000000.iommu: Stage-1: 39-bit VA -> 48-bit IPA
[ 0.412372] arm-smmu 12000000.iommu: Stage-2: 39-bit IPA -> 48-bit PA
[ 0.414488] vdso: 2 pages (1 code @ ffffff8008f67000, 1 data @ ffffff8009e44000)
[ 0.414501] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.416908] atomic_pool_init():622: DMA: preallocated 1024 KiB pool for atomic allocations
[ 0.418774] Registering BPMP clocks…
[ 0.418871] tegra_bpmp_clk_init: clock init ok (638 clks)
[ 0.421002] tegra_powergate_init: DONE
[ 0.421011] DTS File Name: /dvs/git/dirty/git-master_linux/kernel/kernel-4.9/arch/arm64/boot/dts/…/…/…/…/…/…/hardware/nvidia/platform/t18x/quill/kernel-dts/tegra186-quill-p3489-1000-a00-00-base.dts
[ 0.421016] DTB Build time:
[ 0.423167] Tegra reboot handler registered.
[ 0.424707] tegra-pmc c360000.pmc: scratch reg offset dts data not present
[ 0.424777] tegra-pmc: get_secure_pmc_setting: done secure_pmc=0
[ 0.424788] tegra-pmc: ### PMC reset source: TEGRA_SOFTWARE_RESET
[ 0.424791] tegra-pmc: ### PMC reset level: TEGRA_RESET_LEVEL_L1
[ 0.424794] tegra-pmc: ### PMC reset status reg: 0x2d
[ 0.424955] padctrl padctrl.0: Pad control driver tegra-pmc-padctrl registered
[ 0.424966] tegra-pmc c360000.pmc: IO padctrl driver initialized
[ 0.425322] iommu: Adding device 3460000.sdhci to group 0
[ 0.425795] iommu: Adding device 3440000.sdhci to group 1
[ 0.426261] iommu: Adding device 3400000.sdhci to group 2
[ 0.427362] iommu: Adding device 3507000.ahci-sata to group 3
[ 0.427375] arm-smmu: forcing sodev map for 3507000.ahci-sata
[ 0.427826] iommu: Adding device 3160000.i2c to group 4
[ 0.428282] iommu: Adding device c240000.i2c to group 5
[ 0.428640] iommu: Adding device 3180000.i2c to group 6
[ 0.428987] iommu: Adding device 3190000.i2c to group 7
[ 0.429431] iommu: Adding device 31b0000.i2c to group 8
[ 0.429765] iommu: Adding device 31c0000.i2c to group 9
[ 0.430097] iommu: Adding device c250000.i2c to group 10
[ 0.430427] iommu: Adding device 31e0000.i2c to group 11
[ 0.431916] iommu: Adding device 3210000.spi to group 12
[ 0.432268] iommu: Adding device c260000.spi to group 13
[ 0.432599] iommu: Adding device 3240000.spi to group 14
[ 0.433303] iommu: Adding device 3100000.serial to group 15
[ 0.433644] iommu: Adding device 3110000.serial to group 16
[ 0.433972] iommu: Adding device c280000.serial to group 17
[ 0.434299] iommu: Adding device 3130000.serial to group 18
[ 0.435645] iommu: Adding device 2490000.ether_qos to group 19
[ 0.435665] arm-smmu: forcing sodev map for 2490000.ether_qos
[ 0.436348] iommu: Adding device b000000.rtcpu to group 20
[ 0.438566] iommu: Adding device 13e10000.host1x to group 21
[ 0.438891] iommu: Adding device 13e10000.host1x:ctx0 to group 22
[ 0.439211] iommu: Adding device 13e10000.host1x:ctx1 to group 23
[ 0.439517] iommu: Adding device 13e10000.host1x:ctx2 to group 24
[ 0.439827] iommu: Adding device 13e10000.host1x:ctx3 to group 25
[ 0.440134] iommu: Adding device 13e10000.host1x:ctx4 to group 26
[ 0.440461] iommu: Adding device 13e10000.host1x:ctx5 to group 27
[ 0.440776] iommu: Adding device 13e10000.host1x:ctx6 to group 28
[ 0.441090] iommu: Adding device 13e10000.host1x:ctx7 to group 29
[ 0.441611] iommu: Adding device 150c0000.nvcsi to group 30
[ 0.442249] iommu: Adding device 15700000.vi to group 31
[ 0.442605] iommu: Adding device 15600000.isp to group 32
[ 0.443069] iommu: Adding device 15210000.nvdisplay to group 33
[ 0.443667] iommu: Adding device 15340000.vic to group 34
[ 0.443883] iommu: Adding device 154c0000.nvenc to group 35
[ 0.444102] iommu: Adding device 15480000.nvdec to group 36
[ 0.444445] iommu: Adding device 15380000.nvjpg to group 37
[ 0.444658] iommu: Adding device 15500000.tsec to group 38
[ 0.444872] iommu: Adding device 15100000.tsecb to group 39
[ 0.445558] iommu: Adding device 15810000.se to group 40
[ 0.445774] iommu: Adding device 15820000.se to group 41
[ 0.445996] iommu: Adding device 15830000.se to group 42
[ 0.446219] iommu: Adding device 15840000.se to group 43
[ 0.447095] iommu: Adding device smmu_test to group 44
[ 0.447696] mc: mapped MMIO address: 0xffffff800a690000 -> 0x2c10000
[ 0.447707] mc: mapped MMIO address: 0xffffff800a6b0000 -> 0x2c20000
[ 0.447717] mc: mapped MMIO address: 0xffffff800a6d0000 -> 0x2c30000
[ 0.447727] mc: mapped MMIO address: 0xffffff800a7b0000 -> 0x2c40000
[ 0.447737] mc: mapped MMIO address: 0xffffff800a7d0000 -> 0x2c50000
[ 0.447745] nv-tegra-mc 2c10000.mc: No mssnvlink node
[ 0.447755] mc-err: mcerr ops are set to t18x
[ 0.447901] dram-ecc: DRAM ECC enabled-MC_ECC_CONTROL:0x0000000d
[ 0.447924] dram-ecc: SCE-R5 is booted with Camera FW
[ 0.447927] dram-ecc: DRAM ECC interrupts handled in Camera FW
[ 0.449021] iommu: Adding device 3530000.xhci to group 45
[ 0.449033] arm-smmu: forcing sodev map for 3530000.xhci
[ 0.449504] iommu: Adding device 3550000.xudc to group 46
[ 0.449514] arm-smmu: forcing sodev map for 3550000.xudc
[ 0.450237] tegra186-aowake c370000.pmc: WAKE_AOWAKE_CTRL_0 = 3
[ 0.450243] tegra186-aowake c370000.pmc: WAKE_AOWAKE_CNTRL_24(PMU_INT) = 256
[ 0.450822] iommu: Adding device 17000000.gp10b to group 47
[ 0.451857] tegra-reset 5000000.clock: registered 193 resets.
[ 0.454059] iommu: Adding device d000000.bpmp to group 48
[ 0.454269] bpmp: ping status is 0
[ 0.454369] bpmp d000000.bpmp: firmware tag is 4208eb5c85fe47a2235fbc0b8eb87dca
[ 0.456821] bpmp d000000.bpmp: probe ok
[ 0.460006] iommu: Adding device 2600000.dma to group 49
[ 0.461199] GPIO line 445 (sdmmc-wake-support-input) hogged as input
[ 0.461220] GPIO line 446 (sdmmc-wake-support-output) hogged as output/low
[ 0.461245] GPIO line 461 (camera-control-output-low) hogged as output/low
[ 0.461263] GPIO line 456 (camera-control-output-low) hogged as output/low
[ 0.461281] GPIO line 457 (camera-control-output-low) hogged as output/low
[ 0.461300] GPIO line 426 (camera-control-output-low) hogged as output/low
[ 0.461494] gpiochip_setup_dev: registered GPIOs 320 to 511 on device: gpiochip0 (tegra-gpio)
[ 0.465298] gpiochip_setup_dev: registered GPIOs 256 to 319 on device: gpiochip1 (tegra-gpio-aon)
[ 0.466865] iommu: Adding device 10003000.pcie-controller to group 50
[ 0.466881] arm-smmu: forcing sodev map for 10003000.pcie-controller
[ 0.467219] iommu: Adding device sound to group 51
[ 0.467589] iommu: Adding device 3510000.hda to group 52
[ 0.475106] iommu: Adding device c1a0000.aon to group 53
[ 0.475554] vdd-ac-bat: 5000 mV
[ 0.477293] vdd-1v8-ap: 1800 mV
[ 0.501370] random: fast init done
[ 0.518541] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.524632] eventlib_kernel: keventlib is initialized, test id: 0
[ 0.525173] SCSI subsystem initialized
[ 0.525465] libata version 3.00 loaded.
[ 0.525724] usbcore: registered new interface driver usbfs
[ 0.525775] usbcore: registered new interface driver hub
[ 0.525895] usbcore: registered new device driver usb
[ 0.529893] GPIO line 241 (touch-rails) hogged as output/high
[ 0.530138] GPIO line 242 (touch-rails) hogged as output/high
[ 0.530548] gpiochip_setup_dev: registered GPIOs 240 to 255 on device: gpiochip2 (tca9539)
[ 0.531653] GPIO line 228 (lcd-bias-rails) hogged as output/high
[ 0.531995] gpiochip_setup_dev: registered GPIOs 224 to 239 on device: gpiochip3 (tca9539)
[ 0.544407] max77620 4-003c: PMIC Version OTP:0x4D and ES:0x1
[ 0.549917] vddio-ddr: at 1125 mV
[ 0.553759] avdd_dsi_csi: 1200 mV
[ 0.557758] vdd-1v8: 1800 mV
[ 0.561374] vdd-3v3-sys: 3300 mV
[ 0.562105] max77620-sd4: at 600 mV
[ 0.563082] spmic-ldo0: at 1800 mV
[ 0.563910] spmic-ldo1: at 800 mV
[ 0.565363] vddio-3v3: 3300 mV
[ 0.566098] vddio-sdmmc1: 1800 <–> 3300 mV at 3300 mV
[ 0.569453] vdd-rtc: at 800 mV
[ 0.570206] avdd-ts-hv: 1800 <–> 3300 mV at 1800 mV
[ 0.573383] spmic-ldo6: at 1500 mV
[ 0.577365] vdd-pex-1v00: 1000 mV
[ 0.578181] dvdd-pex: 1000 mV
[ 0.578849] GPIO line 221 (spmic_gpio_input) hogged as input
[ 0.579016] GPIO line 222 (spmic_gpio_input) hogged as input
[ 0.579355] gpiochip_setup_dev: registered GPIOs 216 to 223 on device: gpiochip4 (max77620-gpio)
[ 0.581840] max77620 4-003c: max77620 probe successful
[ 0.582345] media: Linux media interface: v0.10
[ 0.582393] Linux video capture interface: v2.00
[ 0.586775] pps_core: LinuxPPS API ver. 1 registered
[ 0.586781] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti giometti@linux.it
[ 0.586802] PTP clock support registered
[ 0.587478] tegra_wdt_t18x 30c0000.watchdog: Tegra WDT init timeout = 120 sec
[ 0.587515] tegra_wdt_t18x 30c0000.watchdog: Registered successfully
[ 0.591716] tegra_fiq_debugger_init: found FIQ source (IRQ 386)
[ 0.597280] trusty trusty: trusty version: Built: 21:17:07 Aug 12 2019
[ 0.597324] trusty trusty: selected api version: 3 (requested 3)
[ 0.598450] Advanced Linux Sound Architecture Driver Initialized.
[ 0.598936] Bluetooth: Core ver 2.22
[ 0.598983] NET: Registered protocol family 31
[ 0.598989] Bluetooth: HCI device and connection manager initialized
[ 0.599000] Bluetooth: HCI socket layer initialized
[ 0.599009] Bluetooth: L2CAP socket layer initialized
[ 0.599034] Bluetooth: SCO socket layer initialized
[ 0.599942] en-vdd-sd: 3300 mV
[ 0.600586] en-vdd-cam: 1800 mV
[ 0.605760] vdd-hdmi: 5000 mV
[ 0.606199] vdd-usb0-5v: 5000 mV
[ 0.606555] vdd-usb1-5v: 5000 mV
[ 0.609767] en-vdd-ts-1v8: 1800 mV
[ 0.613756] en-vdd-ts-hv-3v3: 3300 mV
[ 0.614437] en-vdd-disp-3v3: 3300 mV
[ 0.615122] en-mdm-pwr-3v7: 3700 mV
[ 0.615758] en-vdd-disp-1v8: 1800 mV
[ 0.616368] en-vdd-cam-hv-2v8: 2800 mV
[ 0.617066] en-vdd-cam-1v2: 1200 mV
[ 0.617674] vdd-fan: 5000 mV
[ 0.618022] vdd-3v3: 3300 mV
[ 0.618786] dis-vdd-1v2: 1200 mV
[ 0.619176] en-vdd-vcm-2v8: 2800 mV
[ 0.619766] vdd-usb2-5v: 5000 mV
[ 0.620100] vdd-sys-bl: 3300 mV
[ 0.620689] en-vdd-sys: 1200 mV
[ 0.623977] camchar: rtcpu character device driver loaded
[ 0.624586] gpio tegra-gpio wake53 for gpio=159(X:7)
[ 0.626602] extcon-gpio-states external-connection:extcon@1: Cable state:0, cable id:0
[ 0.628436] clocksource: Switched to clocksource arch_sys_counter
[ 0.669973] VFS: Disk quotas dquot_6.6.0
[ 0.670199] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.670787] nvmap_heap_init: nvmap_heap_init: created heap block cache
[ 0.671733] nvmap: nvmap_select_cache_ops() nvmap cache ops set to roc
[ 0.671742] nvmap_page_pool_init: Total RAM pages: 1750636
[ 0.671748] nvmap_page_pool_init: nvmap page pool size: 218829 pages (854 MB)
[ 0.671949] nvmap_background_zero_thread: PP zeroing thread starting.
[ 0.675918] tegra-gpcdma 2600000.dma: GPC DMA driver register 32 channels
[ 0.687903] thermal thermal_zone0: Registering thermal zone thermal_zone0 for type BCPU-therm
[ 0.688115] thermal thermal_zone1: Registering thermal zone thermal_zone1 for type MCPU-therm
[ 0.688602] thermal thermal_zone2: Registering thermal zone thermal_zone2 for type GPU-therm
[ 0.688716] thermal thermal_zone3: Registering thermal zone thermal_zone3 for type PLL-therm
[ 0.688856] thermal thermal_zone4: Registering thermal zone thermal_zone4 for type Tboard_tegra
[ 0.688987] thermal thermal_zone5: Registering thermal zone thermal_zone5 for type Tdiode_tegra
[ 0.689162] thermal thermal_zone6: Registering thermal zone thermal_zone6 for type PMIC-Die
[ 0.689399] la/ptsa driver initialized.
[ 0.689411] pre_t19x_iso_plat_init(): iso emc max clk=1600000KHz
[ 0.689416] pre_t19x_iso_plat_init(): max_iso_bw=7680000KB
[ 0.690561] NET: Registered protocol family 2
[ 0.692569] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.693265] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 0.694101] TCP: Hash tables configured (established 65536 bind 65536)
[ 0.694163] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[ 0.694289] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[ 0.695253] NET: Registered protocol family 1
[ 0.700395] RPC: Registered named UNIX socket transport module.
[ 0.700402] RPC: Registered udp transport module.
[ 0.700407] RPC: Registered tcp transport module.
[ 0.700411] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.700452] PCI: CLS 0 bytes, default 64
[ 0.700708] Trying to unpack rootfs image as initramfs…
[ 0.890876] Freeing initrd memory: 5432K
[ 0.912730] host1x 13e10000.host1x: initialized
[ 0.914191] hw perfevents: enabled with armv8_pmuv3 PMU driver, 7 counters available
[ 0.918734] audit: initializing netlink subsys (disabled)
[ 0.918775] audit: type=2000 audit(0.808:1): initialized
[ 0.919421] workingset: timestamp_bits=46 max_order=21 bucket_order=0
[ 0.937146] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.940242] ntfs: driver 2.1.32 [Flags: R/W].
[ 0.941975] 9p: Installing v9fs 9p2000 file system support
[ 0.947582] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 240)
[ 0.947730] io scheduler noop registered
[ 0.948347] io scheduler cfq registered (default)
[ 0.953689] gic 2a41000.agic-controller: GIC IRQ controller registered
[ 1.013279] iommu: Adding device aconnect@2a41000:adsp_audio to group 54
[ 1.051872] iommu: Adding device 2993000.adsp to group 55
[ 1.053054] tegra-aconnect aconnect@2a41000: Tegra ACONNECT bus registered
[ 1.055965] tegra-xusb-padctl 3520000.xusb_padctl: VBUS over-current detection enabled
[ 1.059440] tegra-pwm 3280000.pwm: PWM clk can sleep in ops
[ 1.060841] tegra-pwm 3290000.pwm: PWM clk can sleep in ops
[ 1.062256] tegra-pwm 32a0000.pwm: PWM clk can sleep in ops
[ 1.063630] tegra-pwm c340000.pwm: PWM clk can sleep in ops
[ 1.066714] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
[ 1.067818] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
[ 1.068217] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
[ 1.068414] tegra_camera_platform tegra-camera-platform: tegra_camera_probe:camera_platform_driver probe
[ 1.068599] misc tegra_camera_ctrl: tegra_camera_isomgr_register tpg_max_iso = 3916800KBs
[ 1.068608] misc tegra_camera_ctrl: tegra_camera_isomgr_register isp_iso_bw=4687500, vi_iso_bw=1500000, max_bw=4687500
[ 1.071627] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
[ 1.072917] tsec 15500000.tsec: initialized
[ 1.073883] tsec 15100000.tsecb: initialized
[ 1.077243] nvdec 15480000.nvdec: initialized
[ 1.083413] falcon 15340000.vic: initialized
[ 1.084606] falcon 154c0000.nvenc: initialized
[ 1.085674] falcon 15380000.nvjpg: initialized
[ 1.087713] iommu_context_dev 13e10000.host1x:ctx0: initialized (streamid=56)
[ 1.089161] iommu_context_dev 13e10000.host1x:ctx1: initialized (streamid=57)
[ 1.090582] iommu_context_dev 13e10000.host1x:ctx2: initialized (streamid=58)
[ 1.091996] iommu_context_dev 13e10000.host1x:ctx3: initialized (streamid=59)
[ 1.093452] iommu_context_dev 13e10000.host1x:ctx4: initialized (streamid=60)
[ 1.094949] iommu_context_dev 13e10000.host1x:ctx5: initialized (streamid=61)
[ 1.096435] iommu_context_dev 13e10000.host1x:ctx6: initialized (streamid=62)
[ 1.100426] iommu_context_dev 13e10000.host1x:ctx7: initialized (streamid=63)
[ 1.105493] tegradccommon 15200000.dc_common: host1x channel mapped
[ 1.105508] tegradccommon 15200000.dc_common: dc_common syncpt # 1 allocated
[ 1.105553] tegradccommon 15200000.dc_common: dma mapping done
[ 1.107297] tegradc 15210000.nvdisplay: disp0 connected to head1->/host1x/sor1
[ 1.107474] generic_infoframe_type: 0x87
[ 1.107708] tegradc 15210000.nvdisplay: DT parsed successfully
[ 1.107793] tegradc 15210000.nvdisplay: Display dc.ffffff800b690000 registered with id=0
[ 1.116392] tegra-adma 2930000.adma: Tegra210 ADMA driver registered 10 channels
[ 1.117852] tegra-fuse-burn 3820000.efuse:efuse-burn: Fuse burn driver initialized
[ 1.119055] pre_t19x_iso_plat_register(): iso bandwidth 5924200KB is not available, client disp_0
[ 1.119101] pre_t19x_iso_plat_register(): iso bandwidth 3136300KB is not available, client disp_0
[ 1.119237] tegra_nvdisp_bandwidth_register_max_config: max config iso bw = 2787800 KB/s
[ 1.119243] tegra_nvdisp_bandwidth_register_max_config: max config EMC floor = 102000000 Hz
[ 1.119249] tegra_nvdisp_bandwidth_register_max_config: max config hubclk = 49070000 Hz
[ 1.119445] kfuse 3830000.kfuse: initialized
[ 1.119575] tegradc 15210000.nvdisplay: vblank syncpt # 8 for dc 1
[ 1.119590] tegradc 15210000.nvdisplay: vpulse3 syncpt # 9 for dc 1
[ 1.122308] tegra-pmc-iopower pmc-iopower: NO_IOPOWER setting 0x0
[ 1.126703] tegradc 15210000.nvdisplay: hdmi: invalid prod list prod_list_hdmi_board
[ 1.126748] tegradc 15210000.nvdisplay: hdmi: tegra_hdmi_tmds_range_read(bd) failed
[ 1.128232] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 1.133357] console [ttyS0] disabled
[ 1.133410] 3100000.serial: ttyS0 at MMIO 0x3100000 (irq = 36, base_baud = 25500000) is a Tegra
[ 1.133652] tegradc 15210000.nvdisplay: probed
[ 1.151329] console [ttyS0] enabled
[ 1.151835] Console: switching to colour frame buffer device 80x30
[ 1.151877] tegradc 15210000.nvdisplay: fb registered
[ 1.152209] gpio tegra-gpio wake21 for gpio=121(P:1)
[ 1.153659] 3110000.serial: ttyTHS1 at MMIO 0x3110000 (irq = 37, base_baud = 0) is a TEGRA_UART
[ 1.154949] c280000.serial: ttyTHS2 at MMIO 0xc280000 (irq = 38, base_baud = 0) is a TEGRA_UART
[ 1.155190] serial-tegra 3130000.serial: RX in PIO mode
[ 1.156130] 3130000.serial: ttyTHS3 at MMIO 0x3130000 (irq = 39, base_baud = 0) is a TEGRA_UART
[ 1.157444] [drm] Initialized
[ 1.168275] brd: module loaded
[ 1.173461] loop: module loaded
[ 1.173557] tegra_profiler: version: 1.134, samples/io: 47/26
[ 1.173697] tegra_profiler: auth: init
[ 1.174336] nct1008_nct72 7-004c: starting parse dt
[ 1.174477] nct1008_nct72 7-004c: !!!Found deprecated property!!!
[ 1.174489] nct1008_nct72 7-004c: success parsing dt
[ 1.174610] nct1008_nct72 7-004c: success in enabling tmp451 VDD rail
[ 1.174735] THERMAL EST: found 3 subdevs
[ 1.174742] THERMAL EST num_resources: 0
[ 1.174748] [THERMAL EST subdev 0]
[ 1.174755] [THERMAL EST subdev 1]
[ 1.174761] [THERMAL EST subdev 2]
[ 1.175136] thermal thermal_zone7: Registering thermal zone thermal_zone7 for type thermal-fan-est
[ 1.175141] THERMAL EST: thz register success.
[ 1.175320] THERMAL EST: end of probe, return err: 0
[ 1.176407] hisi_sas: driver version v1.6
[ 1.181018] tegra-ahci 3507000.ahci-sata: AHCI 0001.0301 32 slots 2 ports 3 Gbps 0x1 impl platform mode
[ 1.181033] tegra-ahci 3507000.ahci-sata: flags: 64bit ncq sntf pm led pmp pio slum part deso sadm apst
[ 1.182135] scsi host0: tegra_ahci
[ 1.183181] scsi host1: tegra_ahci
[ 1.183404] ata1: SATA max UDMA/133 mmio [mem 0x03507000-0x03508fff] port 0x100 irq 24
[ 1.183411] ata2: DUMMY
[ 1.189836] libphy: Fixed MDIO Bus: probed
[ 1.190408] tun: Universal TUN/TAP device driver, 1.6
[ 1.190413] tun: © 1999-2004 Max Krasnyansky maxk@qualcomm.com
[ 1.191215] e1000e: Intel® PRO/1000 Network Driver - 3.2.6-k
[ 1.191221] e1000e: Copyright© 1999 - 2015 Intel Corporation.
[ 1.191257] igb: Intel® Gigabit Ethernet Network Driver - version 5.4.0-k
[ 1.191262] igb: Copyright © 2007-2014 Intel Corporation.
[ 1.191297] igbvf: Intel® Gigabit Virtual Function Network Driver - version 2.4.0-k
[ 1.191301] igbvf: Copyright © 2009 - 2012 Intel Corporation.
[ 1.191333] Intel® 10GbE PCI Express Linux Network Driver - version 4.6.4
[ 1.191338] Copyright© 1999 - 2017 Intel Corporation.
[ 1.193131] eqos 2490000.ether_qos: can’t get pllrefe_vcoout clk (-2)
[ 1.196964] eqos 2490000.ether_qos: Setting local MAC: 0 4 4b a9 25 b3
[ 1.197117] libphy: dwc_phy: probed
[ 1.197499] bcm54xx_low_power_mode(): put phy in iddq-lp mode
[ 1.199964] PPP generic driver version 2.4.2
[ 1.200144] PPP BSD Compression module registered
[ 1.200151] PPP Deflate Compression module registered
[ 1.200202] PPP MPPE Compression module registered
[ 1.200208] NET: Registered protocol family 24
[ 1.200257] usbcore: registered new interface driver r8152
[ 1.200292] usbcore: registered new interface driver asix
[ 1.200330] usbcore: registered new interface driver ax88179_178a
[ 1.200360] usbcore: registered new interface driver cdc_ether
[ 1.200390] usbcore: registered new interface driver net1080
[ 1.200451] usbcore: registered new interface driver cdc_subset
[ 1.200485] usbcore: registered new interface driver zaurus
[ 1.200530] usbcore: registered new interface driver cdc_ncm
[ 1.200741] VFIO - User Level meta-driver version: 0.3
[ 1.202363] ehci_hcd: USB 2.0 ‘Enhanced’ Host Controller (EHCI) Driver
[ 1.202381] ehci-pci: EHCI PCI platform driver
[ 1.202416] ohci_hcd: USB 1.1 ‘Open’ Host Controller (OHCI) Driver
[ 1.202426] ohci-pci: OHCI PCI platform driver
[ 1.202459] ohci-platform: OHCI generic platform driver
[ 1.206941] nct1008_nct72 7-004c: LOC shutdown limit 110
[ 1.207505] nct1008_nct72 7-004c: EXT shutdown limit 110
[ 1.207935] tegra-xusb 3530000.xhci: USB2 port 0 has OTG_CAP
[ 1.209699] tegra-xusb 3530000.xhci: extcon 0: ffffffc1ac5d3000 id
[ 1.210419] usbcore: registered new interface driver uas
[ 1.210463] usbcore: registered new interface driver usb-storage
[ 1.210542] usbcore: registered new interface driver usbserial
[ 1.211652] tegra-xusb 3530000.xhci: Firmware timestamp: 2018-12-26 10:30:04 UTC, Version: 55.12 release
[ 1.215277] tegra-xudc-new 3550000.xudc: device count: 1
[ 1.216529] tegra-xudc-new 3550000.xudc: vbus state: 0
[ 1.216943] tegra-xudc-new 3550000.xudc: entering ELPG
[ 1.217143] mousedev: PS/2 mouse device common for all mice
[ 1.217216] usbcore: registered new interface driver xpad
[ 1.219093] tegra-xudc-new 3550000.xudc: entering ELPG done
[ 1.236524] tegra-xusb 3530000.xhci: xHCI Host Controller
[ 1.236545] tegra-xusb 3530000.xhci: new USB bus registered, assigned bus number 1
[ 1.237404] tegra-xusb 3530000.xhci: hcc params 0x0184fd25 hci version 0x100 quirks 0x00050810
[ 1.237445] tegra-xusb 3530000.xhci: irq 70, io mem 0x03530000
[ 1.237699] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 1.237707] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.237714] usb usb1: Product: xHCI Host Controller
[ 1.237720] usb usb1: Manufacturer: Linux 4.9.140-tegra xhci-hcd
[ 1.237725] usb usb1: SerialNumber: 3530000.xhci
[ 1.238212] hub 1-0:1.0: USB hub found
[ 1.238260] hub 1-0:1.0: 4 ports detected
[ 1.238687] tegra-xusb 3530000.xhci: xHCI Host Controller
[ 1.238700] tegra-xusb 3530000.xhci: new USB bus registered, assigned bus number 2
[ 1.238923] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[ 1.238931] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.238937] usb usb2: Product: xHCI Host Controller
[ 1.238942] usb usb2: Manufacturer: Linux 4.9.140-tegra xhci-hcd
[ 1.238948] usb usb2: SerialNumber: 3530000.xhci
[ 1.239363] hub 2-0:1.0: USB hub found
[ 1.239409] hub 2-0:1.0: 3 ports detected
[ 1.240371] nct1008_nct72 7-004c: initial LOC temp: 28000
[ 1.240703] nct1008_nct72 7-004c: initial EXT temp: 29250
[ 1.241338] nct1008_nct72 7-004c: hi_b:-11, lo_b:0
[ 1.241715] gpio tegra-gpio-aon wake35 for gpio=16(V:0)
[ 1.241923] nct1008_nct72 7-004c: nct1008_probe: initialized
[ 1.315375] max77686-rtc max77620-rtc: rtc core: registered max77620-rtc as rtc0
[ 1.316565] Wake73 for irq=42
[ 1.317823] rtc rtc1: alarm rtc device
[ 1.317870] tegra_rtc c2a0000.rtc: rtc core: registered c2a0000.rtc as rtc1
[ 1.317894] tegra_rtc c2a0000.rtc: Tegra internal Real Time Clock
[ 1.318079] i2c /dev entries driver
[ 1.319814] ov5693 2-0036: probing v4l2 sensor.
[ 1.320916] ov5693 2-0036: tegracam sensor driver:ov5693_v2.0.6
[ 1.340651] usb usb1: usb_suspend_both: status 0
[ 1.501196] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 1.660498] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 1.660813] ata1.00: ATA-10: Micron_1100_MTFDDAK512TBN, M0MU031, max UDMA/133
[ 1.660822] ata1.00: 1000215216 sectors, multi 16: LBA48 NCQ (depth 31/32)
[ 1.662157] ata1.00: configured for UDMA/133
[ 1.672854] scsi 0:0:0:0: Direct-Access ATA Micron_1100_MTFD U031 PQ: 0 ANSI: 5
[ 1.705178] ata1.00: Enabling discard_zeroes_data
[ 1.705229] sd 0:0:0:0: [sda] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
[ 1.705466] sd 0:0:0:0: [sda] Write Protect is off
[ 1.705477] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 1.705550] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA
[ 1.706004] ata1.00: Enabling discard_zeroes_data
[ 1.706839] sda: sda1
[ 1.707302] ata1.00: Enabling discard_zeroes_data
[ 1.707734] sd 0:0:0:0: [sda] Attached SCSI disk
[ 1.891717] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 2.095788] max77620-power max20024-power: Event recorder REG_NVERC : 0x0
[ 2.102042] parse_throttle_dt_data: Num cap clks = 4
[ 2.102051] parse_throttle_dt_data: clk=mcpu type=1
[ 2.102056] parse_throttle_dt_data: clk=bcpu type=2
[ 2.102066] parse_throttle_dt_data: clk=gpu type=4
[ 2.102071] parse_throttle_dt_data: clk=emc type=3
[ 2.103103] tegra_throttle_probe: probe successful. #cdevs=4
[ 2.103890] FAN dev name: pwm-fan
[ 2.103951] FAN:gpio request success.
[ 2.103959] FAN: can’t find tach_gpio
[ 2.103984] pwm_fan_driver pwm-fan: cap state:7, cap pwm:255
[ 2.104193] pwm_fan_driver pwm-fan: got pwm for fan. polarity is inversed
[ 2.104197] pwm_fan_driver pwm-fan: tach period: 1000
[ 2.104587] pwm_fan_driver pwm-fan: index 0: pwm=0, rpm=0, rru=40, rrd=40, state:2
[ 2.104597] pwm_fan_driver pwm-fan: index 1: pwm=80, rpm=1000, rru=2, rrd=2, state:2
[ 2.104604] pwm_fan_driver pwm-fan: index 2: pwm=120, rpm=2000, rru=1, rrd=1, state:2
[ 2.104609] pwm_fan_driver pwm-fan: index 3: pwm=160, rpm=3000, rru=1, rrd=1, state:2
[ 2.104615] pwm_fan_driver pwm-fan: index 4: pwm=255, rpm=4000, rru=1, rrd=1, state:3
[ 2.104620] pwm_fan_driver pwm-fan: index 5: pwm=255, rpm=5000, rru=1, rrd=1, state:3
[ 2.104625] pwm_fan_driver pwm-fan: index 6: pwm=255, rpm=6000, rru=1, rrd=1, state:3
[ 2.104630] pwm_fan_driver pwm-fan: index 7: pwm=255, rpm=7000, rru=1, rrd=1, state:4
[ 2.104645] pwm_fan_driver pwm-fan: index 8: pwm=255, rpm=10000, rru=1, rrd=1, state:4
[ 2.104652] pwm_fan_driver pwm-fan: index 9: pwm=255, rpm=11000, rru=1, rrd=1, state:4
[ 2.106582] device-mapper: uevent: version 1.0.3
[ 2.107464] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[ 2.110356] cpufreq: platform driver Initialization: pass
[ 2.111276] cpuidle: Initializing cpuidle driver init for Denver cluster
[ 2.111406] cpuidle: Initializing cpuidle driver init for A57 cluster
[ 2.112354] sdhci: Secure Digital Host Controller Interface driver
[ 2.112358] sdhci: Copyright© Pierre Ossman
[ 2.112361] sdhci-pltfm: SDHCI platform and OF driver helper
[ 2.114016] sdhci-tegra 3460000.sdhci: Client registration for eMC Successful
[ 2.115918] CMDQ: cmdq_platfm_init successful
[ 2.117078] CQE: CMD_TIMING bit set for R1B DCMD
[ 2.119424] sdhci-tegra 3440000.sdhci: Client registration for eMC Successful
[ 2.125677] sdhci-tegra 3400000.sdhci: Got CD GPIO
[ 2.125740] sdhci-tegra 3400000.sdhci: Got WP GPIO
[ 2.127174] sdhci-tegra 3400000.sdhci: Client registration for eMC Successful
[ 2.129042] sdhci-tegra 3400000.sdhci: wakeup init done, cdirq 257
[ 2.133245] tegra-se-elp 3ad0000.se_elp: tegra_se_elp_probe: complete
[ 2.134357] tegra-se-nvhost 15810000.se: initialized
[ 2.135455] tegra-se-nvhost 15810000.se: tegra_se_probe: complete
[ 2.136551] tegra-se-nvhost 15820000.se: initialized
[ 2.137666] tegra-se-nvhost 15820000.se: tegra_se_probe: complete
[ 2.138603] tegra-se-nvhost 15830000.se: initialized
[ 2.139271] tegra-se-nvhost 15830000.se: tegra_se_probe: complete
[ 2.140181] tegra-se-nvhost 15840000.se: initialized
[ 2.141289] tegra-se-nvhost 15840000.se: tegra_se_probe: complete
[ 2.141556] hidraw: raw HID events driver © Jiri Kosina
[ 2.145698] usbcore: registered new interface driver usbhid
[ 2.145703] usbhid: USB HID core driver
[ 2.153881] tegra186-cam-rtcpu b000000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[ 2.156237] tegra-ivc ivc-b000000.rtcpu: region 0: iova=0xbfef0000-0xbfefdb7f size=56192
[ 2.156532] tegra-ivc ivc-b000000.rtcpu:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[ 2.156678] tegra-ivc ivc-b000000.rtcpu:i2c@480: i2c: ver=0 grp=1 RX[8x128]=0x1900-0x1d80 TX[8x128]=0x1d80-0x2200
[ 2.156907] tegra-ivc ivc-b000000.rtcpu:vinotify@12c0: vinotify: ver=0 grp=1 RX[64x128]=0x2200-0x4280 TX[64x128]=0x4280-0x6300
[ 2.157819] tegra-ivc ivc-b000000.rtcpu:mods@32c0: mods: ver=0 grp=1 RX[1x64]=0x6300-0x63c0 TX[1x64]=0x63c0-0x6480
[ 2.157891] tegra-ivc ivc-b000000.rtcpu:ivccontrol@52c0: ivccontrol: ver=0 grp=1 RX[16x320]=0x6480-0x7900 TX[16x320]=0x7900-0x8d80
[ 2.157947] tegra-ivc ivc-b000000.rtcpu:ivccapture@72c0: ivccapture: ver=0 grp=1 RX[16x64]=0x8d80-0x9200 TX[16x64]=0x9200-0x9680
[ 2.157995] tegra-ivc ivc-b000000.rtcpu:dbg@7c00: dbg: ver=0 grp=1 RX[1x384]=0x9680-0x9880 TX[1x384]=0x9880-0x9a80
[ 2.158094] tegra-ivc ivc-b000000.rtcpu:dbg@7e00: dbg: ver=0 grp=1 RX[1x8192]=0x9a80-0xbb00 TX[1x8192]=0xbb00-0xdb80
[ 2.158422] tegra186-cam-rtcpu b000000.rtcpu: using cam RTCPU IRQ (51)
[ 2.158425] tegra186-cam-rtcpu b000000.rtcpu: tegra_camrtc_mon_create is successful
[ 2.158940] tegra186-cam-rtcpu b000000.rtcpu: firmware version cpu=sce cmd=5 sha1=3ecb1dca25fa98771f988f238c025334776c9a96
[ 2.160470] mmc0: SDHCI controller on 3460000.sdhci [3460000.sdhci] using ADMA 64-bit with 64 bit addr
[ 2.160685] tegra_aon c1a0000.aon: tegra aon driver probe OK
[ 2.161191] tegra186-aondbg aondbg: aondbg driver probe() OK
[ 2.161393] register_ari_mca_banks: Registered MCA ROC:IOB
[ 2.161466] register_ari_mca_banks: Registered MCA ROC:CCE
[ 2.161807] tegra18-bridge 2390000.axi2apb: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.161810] tegra18-bridge 2390000.axi2apb: enabled timeout = 16684000
[ 2.161812] tegra18-bridge 2390000.axi2apb: bridge probed OK
[ 2.161865] tegra18-bridge 23a0000.axi2apb: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.161868] tegra18-bridge 23a0000.axi2apb: enabled timeout = 16684000
[ 2.161870] tegra18-bridge 23a0000.axi2apb: bridge probed OK
[ 2.161919] tegra18-bridge 23b0000.axi2apb: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.161922] tegra18-bridge 23b0000.axi2apb: enabled timeout = 16684000
[ 2.161924] tegra18-bridge 23b0000.axi2apb: bridge probed OK
[ 2.161974] tegra18-bridge 23c0000.axi2apb: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.161976] tegra18-bridge 23c0000.axi2apb: enabled timeout = 16684000
[ 2.161978] tegra18-bridge 23c0000.axi2apb: bridge probed OK
[ 2.162031] tegra18-bridge 23d0000.axi2apb: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162034] tegra18-bridge 23d0000.axi2apb: enabled timeout = 16684000
[ 2.162036] tegra18-bridge 23d0000.axi2apb: bridge probed OK
[ 2.162085] tegra18-bridge 2100000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162089] tegra18-bridge 2100000.axip2p: enabled timeout = 16684000
[ 2.162091] tegra18-bridge 2100000.axip2p: bridge probed OK
[ 2.162141] tegra18-bridge 2110000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162146] tegra18-bridge 2110000.axip2p: enabled timeout = 16684000
[ 2.162148] tegra18-bridge 2110000.axip2p: bridge probed OK
[ 2.162199] tegra18-bridge 2120000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162203] tegra18-bridge 2120000.axip2p: enabled timeout = 16684000
[ 2.162205] tegra18-bridge 2120000.axip2p: bridge probed OK
[ 2.162254] tegra18-bridge 2130000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162258] tegra18-bridge 2130000.axip2p: enabled timeout = 16684000
[ 2.162260] tegra18-bridge 2130000.axip2p: bridge probed OK
[ 2.162311] tegra18-bridge 2140000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162314] tegra18-bridge 2140000.axip2p: enabled timeout = 16684000
[ 2.162316] tegra18-bridge 2140000.axip2p: bridge probed OK
[ 2.162365] tegra18-bridge 2150000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162369] tegra18-bridge 2150000.axip2p: enabled timeout = 16684000
[ 2.162371] tegra18-bridge 2150000.axip2p: bridge probed OK
[ 2.162421] tegra18-bridge 2160000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162426] tegra18-bridge 2160000.axip2p: enabled timeout = 16684000
[ 2.162428] tegra18-bridge 2160000.axip2p: bridge probed OK
[ 2.162480] tegra18-bridge 2170000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162484] tegra18-bridge 2170000.axip2p: enabled timeout = 16684000
[ 2.162486] tegra18-bridge 2170000.axip2p: bridge probed OK
[ 2.162544] tegra18-bridge 2180000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162548] tegra18-bridge 2180000.axip2p: enabled timeout = 16684000
[ 2.162550] tegra18-bridge 2180000.axip2p: bridge probed OK
[ 2.162600] tegra18-bridge 2190000.axip2p: axi_cbb clk rate = 172 MHZ, timeout = 97000 useconds
[ 2.162604] tegra18-bridge 2190000.axip2p: enabled timeout = 16684000
[ 2.162606] tegra18-bridge 2190000.axip2p: bridge probed OK
[ 2.162665] **** A57 ECC: Enabled
[ 2.162667] tegra18_a57_serr_init: on CPU 5 a A57 Core
[ 2.162734] tegra18x_actmon d230000.actmon: in actmon_register()…
[ 2.163189] tegra18x_actmon d230000.actmon: initialization Completed for the device mc_all
[ 2.163615] hw perfevents: enabled with denver15_uncore_pmu PMU driver, 3 counters available
[ 2.163736] denver_knobs_init:MTS_VERSION:43068234
[ 2.164475] mmc1: SDHCI controller on 3440000.sdhci [3440000.sdhci] using ADMA 64-bit with 64 bit addr
[ 2.165454] nvpmodel: initialized successfully
[ 2.167176] trusty-virtio trusty:virtio: initializing
[ 2.167292] trusty_ipc virtio0: vring0: va(pa) ffffffc1a6560000(0) qsz 32 notifyid 1
[ 2.167298] trusty_ipc virtio0: vring1: va(pa) ffffffc1a6562000(0) qsz 32 notifyid 2
[ 2.167365] trusty-virtio trusty:virtio: initializing done
[ 2.167464] trusty_ipc virtio0: is online
[ 2.169837] usbcore: registered new interface driver snd-usb-audio
[ 2.177128] input: tegra-hda HDMI/DP,pcm=3 as /devices/3510000.hda/sound/card0/input0
[ 2.177299] input: tegra-hda HDMI/DP,pcm=7 as /devices/3510000.hda/sound/card0/input1
[ 2.178460] gpio tegra-gpio wake71 for gpio=125(P:5)
[ 2.178587] mmc2: SDHCI controller on 3400000.sdhci [3400000.sdhci] using ADMA 64-bit with 64 bit addr
[ 2.206273] mmc0: mmc_decode_ext_csd: CMDQ supported: depth: 31, cmdq_support: 1
[ 2.220677] mmc0: periodic cache flush enabled
[ 2.220705] mmc0: new HS400 Enhanced strobe MMC card at address 0001
[ 2.225276] mmcblk0: mmc0:0001 032G34 29.1 GiB
[ 2.225531] mmcblk0boot0: mmc0:0001 032G34 partition 1 8.00 MiB
[ 2.225766] mmcblk0boot1: mmc0:0001 032G34 partition 2 8.00 MiB
[ 2.226008] mmcblk0rpmb: mmc0:0001 032G34 partition 3 4.00 MiB
[ 2.229895] mmcblk0: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 p14 p15 p16 p17 p18 p19 p20 p21 p22 p23 p24 p25 p26 p27 p28 p29 p30 p31
[ 2.281474] OPE platform probe
[ 2.281587] OPE platform probe successful
[ 2.288465] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[ 2.290491] tegra-pcie 10003000.pcie-controller: link 2 down, ignoring
[ 2.294648] tegra-asoc: sound: This is a dummy codec
[ 2.359205] tegra-asoc: sound: ADMAIF1 <-> ADMAIF1 mapping ok
[ 2.359303] tegra-asoc: sound: ADMAIF2 <-> ADMAIF2 mapping ok
[ 2.359393] tegra-asoc: sound: ADMAIF3 <-> ADMAIF3 mapping ok
[ 2.359472] tegra-asoc: sound: ADMAIF4 <-> ADMAIF4 mapping ok
[ 2.359648] tegra-asoc: sound: ADMAIF5 <-> ADMAIF5 mapping ok
[ 2.359729] tegra-asoc: sound: ADMAIF6 <-> ADMAIF6 mapping ok
[ 2.359811] tegra-asoc: sound: ADMAIF7 <-> ADMAIF7 mapping ok
[ 2.359893] tegra-asoc: sound: ADMAIF8 <-> ADMAIF8 mapping ok
[ 2.359972] tegra-asoc: sound: ADMAIF9 <-> ADMAIF9 mapping ok
[ 2.360057] tegra-asoc: sound: ADMAIF10 <-> ADMAIF10 mapping ok
[ 2.360140] tegra-asoc: sound: ADMAIF11 <-> ADMAIF11 mapping ok
[ 2.360222] tegra-asoc: sound: ADMAIF12 <-> ADMAIF12 mapping ok
[ 2.360308] tegra-asoc: sound: ADMAIF13 <-> ADMAIF13 mapping ok
[ 2.360397] tegra-asoc: sound: ADMAIF14 <-> ADMAIF14 mapping ok
[ 2.360519] tegra-asoc: sound: ADMAIF15 <-> ADMAIF15 mapping ok
[ 2.360635] tegra-asoc: sound: ADMAIF16 <-> ADMAIF16 mapping ok
[ 2.360725] tegra-asoc: sound: ADMAIF17 <-> ADMAIF17 mapping ok
[ 2.360811] tegra-asoc: sound: ADMAIF18 <-> ADMAIF18 mapping ok
[ 2.360906] tegra-asoc: sound: ADMAIF19 <-> ADMAIF19 mapping ok
[ 2.360988] tegra-asoc: sound: ADMAIF20 <-> ADMAIF20 mapping ok
[ 2.388096] u32 classifier
[ 2.388101] Actions configured
[ 2.388211] Initializing XFRM netlink socket
[ 2.389943] NET: Registered protocol family 10
[ 2.391368] NET: Registered protocol family 17
[ 2.391384] NET: Registered protocol family 15
[ 2.391603] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 2.391692] Bluetooth: RFCOMM socket layer initialized
[ 2.391709] Bluetooth: RFCOMM ver 1.11
[ 2.391715] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[ 2.391722] Bluetooth: HIDP socket layer initialized
[ 2.391744] 9pnet: Installing 9P2000 support
[ 2.391786] Key type dns_resolver registered
[ 2.392024] Registered cp15_barrier emulation handler
[ 2.392041] Registered setend emulation handler
[ 2.393680] registered taskstats version 1
[ 2.397682] tegra-pcie 10003000.pcie-controller: PCI host bridge to bus 0000:00
[ 2.397691] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 2.397696] pci_bus 0000:00: root bus resource [mem 0x40100000-0x47ffffff]
[ 2.397702] pci_bus 0000:00: root bus resource [mem 0x48000000-0x7fffffff pref]
[ 2.397707] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.397736] pci 0000:00:01.0: [10de:10e5] type 01 class 0x060400
[ 2.397828] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.397971] iommu: Adding device 0000:00:01.0 to group 56
[ 2.397981] arm-smmu: forcing sodev map for 0000:00:01.0
[ 2.398074] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2.398288] pci 0000:01:00.0: [11f8:8536] type 01 class 0x060400
[ 2.399542] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[ 2.401292] iommu: Adding device 0000:01:00.0 to group 57
[ 2.401299] arm-smmu: forcing sodev map for 0000:01:00.0
[ 2.401507] pci 0000:01:00.1: [11f8:8536] type 00 class 0x068000
[ 2.401608] isp 15600000.isp: initialized
[ 2.402009] pci 0000:01:00.1: reg 0x10: [mem 0x00000000-0x003fffff 64bit pref]
[ 2.402351] pci 0000:01:00.1: reg 0x18: [mem 0x00000000-0x03ffffff 64bit pref]
[ 2.404919] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold
[ 2.405392] iommu: Adding device 0000:01:00.1 to group 58
[ 2.405402] arm-smmu: forcing sodev map for 0000:01:00.1
[ 2.407292] nvcsi 150c0000.nvcsi: initialized
[ 2.413050] gpio tegra-gpio-aon wake29 for gpio=56(FF:0)
[ 2.413126] gpio tegra-gpio-aon wake67 for gpio=57(FF:1)
[ 2.413201] gpio tegra-gpio-aon wake68 for gpio=58(FF:2)
[ 2.413361] input: gpio-keys as /devices/gpio-keys/input/input2
[ 2.414616] tegra-vi4 15700000.vi: initialized
[ 2.415793] tegra-vi4 15700000.vi: subdev 150c0000.nvcsi–1 bound
[ 2.415850] tegra-vi4 15700000.vi: subdev ov5693 2-0036 bound
[ 2.416918] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2.418571] pci 0000:02:00.0: [11f8:8536] type 01 class 0x060400
[ 2.421109] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
[ 2.422877] iommu: Adding device 0000:02:00.0 to group 59
[ 2.422909] arm-smmu: forcing sodev map for 0000:02:00.0
[ 2.423106] pci 0000:02:01.0: [11f8:8536] type 01 class 0x060400
[ 2.426410] pci 0000:02:01.0: PME# supported from D0 D3hot D3cold
[ 2.426926] iommu: Adding device 0000:02:01.0 to group 60
[ 2.426931] arm-smmu: forcing sodev map for 0000:02:01.0
[ 2.428470] pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2.428549] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2.429122] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[ 2.430562] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[ 2.430617] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 04
[ 2.430679] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 04
[ 2.431506] pci 0000:00:01.0: BAR 15: assigned [mem 0x48000000-0x4dffffff 64bit pref]
[ 2.431512] pci 0000:00:01.0: BAR 14: assigned [mem 0x40100000-0x404fffff]
[ 2.431517] pci 0000:00:01.0: BAR 13: assigned [io 0x1000-0x2fff]
[ 2.431527] pci 0000:01:00.1: BAR 2: assigned [mem 0x48000000-0x4bffffff 64bit pref]
[ 2.431674] pci 0000:01:00.1: BAR 0: assigned [mem 0x4c000000-0x4c3fffff 64bit pref]
[ 2.432338] pci 0000:01:00.0: BAR 14: assigned [mem 0x40100000-0x404fffff]
[ 2.432353] pci 0000:01:00.0: BAR 15: assigned [mem 0x4c400000-0x4c7fffff 64bit pref]
[ 2.432358] pci 0000:01:00.0: BAR 13: assigned [io 0x1000-0x2fff]
[ 2.432367] pci 0000:02:00.0: BAR 14: assigned [mem 0x40100000-0x402fffff]
[ 2.432373] pci 0000:02:00.0: BAR 15: assigned [mem 0x4c400000-0x4c5fffff 64bit pref]
[ 2.432378] pci 0000:02:01.0: BAR 14: assigned [mem 0x40300000-0x404fffff]
[ 2.432384] pci 0000:02:01.0: BAR 15: assigned [mem 0x4c600000-0x4c7fffff 64bit pref]
[ 2.432388] pci 0000:02:00.0: BAR 13: assigned [io 0x1000-0x1fff]
[ 2.432393] pci 0000:02:01.0: BAR 13: assigned [io 0x2000-0x2fff]
[ 2.432401] pci 0000:02:00.0: PCI bridge to [bus 03]
[ 2.432418] pci 0000:02:00.0: bridge window [io 0x1000-0x1fff]
[ 2.432497] pci 0000:02:00.0: bridge window [mem 0x40100000-0x402fffff]
[ 2.432657] pci 0000:02:00.0: bridge window [mem 0x4c400000-0x4c5fffff 64bit pref]
[ 2.432909] pci 0000:02:01.0: PCI bridge to [bus 04]
[ 2.432926] pci 0000:02:01.0: bridge window [io 0x2000-0x2fff]
[ 2.432996] pci 0000:02:01.0: bridge window [mem 0x40300000-0x404fffff]
[ 2.433157] pci 0000:02:01.0: bridge window [mem 0x4c600000-0x4c7fffff 64bit pref]
[ 2.433409] pci 0000:01:00.0: PCI bridge to [bus 02-04]
[ 2.433425] pci 0000:01:00.0: bridge window [io 0x1000-0x2fff]
[ 2.433472] pci 0000:01:00.0: bridge window [mem 0x40100000-0x404fffff]
[ 2.433652] pci 0000:01:00.0: bridge window [mem 0x4c400000-0x4c7fffff 64bit pref]
[ 2.433933] pci 0000:00:01.0: PCI bridge to [bus 01-04]
[ 2.433937] pci 0000:00:01.0: bridge window [io 0x1000-0x2fff]
[ 2.433961] pci 0000:00:01.0: bridge window [mem 0x40100000-0x404fffff]
[ 2.433967] pci 0000:00:01.0: bridge window [mem 0x48000000-0x4dffffff 64bit pref]
[ 2.434645] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
[ 2.434650] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[ 2.434653] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt
[ 2.434657] pci 0000:02:01.0: Signaling PME through PCIe PME interrupt
[ 2.434660] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt
[ 2.434667] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded
[ 2.434824] aer 0000:00:01.0:pcie002: service driver aer loaded
[ 2.445211] tegra_rtc c2a0000.rtc: setting system clock to 2000-01-05 18:25:51 UTC (947096751)
[ 2.446826] mmcblk mmc0:0001: Card claimed for testing.
[ 2.473590] bpmp: mounted debugfs mirror
[ 2.475678] bwmgr: missing cdev-type property
[ 2.482816] spmic-ldo0: disabling
[ 2.483009] spmic-ldo1: disabling
[ 2.483164] en-vdd-sd: disabling
[ 2.483168] en-vdd-cam: disabling
[ 2.483171] vdd-usb0-5v: disabling
[ 2.483175] vdd-usb1-5v: disabling
[ 2.483179] en-vdd-disp-3v3: disabling
[ 2.483182] en-mdm-pwr-3v7: disabling
[ 2.483205] en-vdd-disp-1v8: disabling
[ 2.483208] en-vdd-cam-hv-2v8: disabling
[ 2.483211] en-vdd-cam-1v2: disabling
[ 2.483214] vdd-fan: disabling
[ 2.483217] vdd-3v3: disabling
[ 2.483221] en-vdd-vcm-2v8: disabling
[ 2.483224] vdd-usb2-5v: disabling
[ 2.483227] vdd-sys-bl: disabling
[ 2.483231] en-vdd-sys: disabling
[ 2.483235] ALSA device list:
[ 2.483238] #0: tegra-hda at 0x3518000 irq 383
[ 2.483241] #1: tegra-snd-t186ref-mobile-rt565x
[ 2.485303] Freeing unused kernel memory: 8576K
[ 2.517741] L4T-INITRD Build DATE: Wed Jan 5 18:25:51 UTC 2000
[ 2.523909] Root device found: mmcblk0p1
[ 2.526349] Found dev node: /dev/mmcblk0p1
[ 2.553926] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[ 2.555200] Rootfs mounted over mmcblk0p1
[ 2.592667] Switching from initrd to actual rootfs
[ 2.743572] systemd[1]: System time before build time, advancing clock.
[ 2.789255] ip_tables: © 2000-2006 Netfilter Core Team
[ 2.836569] cgroup: cgroup2: unknown option “nsdelegate”
[ 2.867035] systemd[1]: systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[ 2.870446] systemd[1]: Detected architecture arm64.
[ 2.963650] systemd[1]: Set hostname to .
[ 3.047902] systemd[1]: File /lib/systemd/system/systemd-journald.service:36 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[ 3.047933] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ 3.175437] systemd[1]: /etc/systemd/system/rc-local.service:11: Support for option SysVStartPriority= has been removed and it is ignored
[ 3.249932] random: systemd: uninitialized urandom read (16 bytes read)
[ 3.249994] systemd[1]: Reached target Swap.
[ 3.250436] random: systemd: uninitialized urandom read (16 bytes read)
[ 3.250756] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[ 3.250854] random: systemd: uninitialized urandom read (16 bytes read)
[ 3.250896] systemd[1]: Reached target User and Group Name Lookups.
[ 3.254712] systemd[1]: Created slice User and Session Slice.
[ 3.366677] EXT4-fs (mmcblk0p1): re-mounted. Opts: (null)
[ 3.605441] systemd-journald[2219]: Received request to flush runtime journal from PID 1
[ 4.237417] using random self ethernet address
[ 4.241917] using random host ethernet address
[ 4.282294] Mass Storage Function, version: 2009/09/11
[ 4.282301] LUN: removable file: (no medium)
[ 4.296904] using random self ethernet address
[ 4.296908] using random host ethernet address
[ 4.345172] rndis0: HOST MAC 6a:16:63:88:e8:fc
[ 4.345891] rndis0: MAC 6a:16:63:88:e8:fd
[ 4.348765] usb0: HOST MAC 6a:16:63:88:e8:fe
[ 4.348812] usb0: MAC 6a:16:63:88:e8:ff
[ 4.348837] tegra-xudc-new 3550000.xudc: exiting ELPG
[ 4.350511] tegra-xudc-new 3550000.xudc: exiting ELPG done
[ 4.350534] tegra-xudc-new 3550000.xudc: ep 0 (type: 0, dir: out) enabled
[ 4.356632] tegra-xudc-new 3550000.xudc: entering ELPG
[ 4.358060] tegra-xudc-new 3550000.xudc: entering ELPG done
[ 4.358981] tegra-xudc-new 3550000.xudc: exiting ELPG
[ 4.376156] tegra-xudc-new 3550000.xudc: exiting ELPG done
[ 4.376185] tegra-xudc-new 3550000.xudc: entering ELPG
[ 4.377568] tegra-xudc-new 3550000.xudc: entering ELPG done
[ 4.379733] Wake76 for irq=199
[ 4.379740] Wake77 for irq=199
[ 4.379743] Wake78 for irq=199
[ 4.379745] Wake79 for irq=199
[ 4.379748] Wake80 for irq=199
[ 4.379750] Wake81 for irq=199
[ 4.379752] Wake82 for irq=199
[ 4.379798] tegra-xusb 3530000.xhci: Upgrade port 0 to USB3.0
[ 4.379804] tegra-xusb 3530000.xhci: Upgrade port 1 to USB3.0
[ 4.393392] l4tbr0: port 1(rndis0) entered blocking state
[ 4.393401] l4tbr0: port 1(rndis0) entered disabled state
[ 4.393592] device rndis0 entered promiscuous mode
[ 4.442263] IPv6: ADDRCONF(NETDEV_UP): rndis0: link is not ready
[ 4.450078] l4tbr0: port 2(usb0) entered blocking state
[ 4.450083] l4tbr0: port 2(usb0) entered disabled state
[ 4.450242] device usb0 entered promiscuous mode
[ 4.462342] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
[ 4.480818] usb usb2: usb_suspend_both: status 0
[ 4.513193] random: crng init done
[ 4.516616] random: 7 urandom warning(s) missed due to ratelimiting
[ 4.733185] nvgpu: 17000000.gp10b railgate_enable_store:283 [INFO] railgate is disabled.
[ 4.751047] switchtec switchtec0: Management device registered.
[ 4.751105] switchtec: loaded.
[ 4.753762] ntb_hw_switchtec: loading out-of-tree module taints kernel.
[ 4.762515] switchtec switchtec0: Partitions: 3
[ 4.767860] switchtec switchtec0: tpart_vec=0000FFFFFFFFFFFD, self_partition=1, epmap=0000000000000003
[ 5.004724] switchtec switchtec0: NTB device registered
[ 5.221839] zram: Added device: zram0
[ 5.224979] zram: Added device: zram1
[ 5.229328] zram: Added device: zram2
[ 5.233259] zram: Added device: zram3
[ 5.237108] zram: Added device: zram4
[ 5.237607] zram: Added device: zram5
[ 5.265298] zram0: detected capacity change from 0 to 598745088
[ 5.311357] Adding 584708k swap on /dev/zram0. Priority:5 extents:1 across:584708k SS
[ 5.336630] zram1: detected capacity change from 0 to 598745088
[ 5.420569] Adding 584708k swap on /dev/zram1. Priority:5 extents:1 across:584708k SS
[ 5.435498] zram2: detected capacity change from 0 to 598745088
[ 5.447972] Adding 584708k swap on /dev/zram2. Priority:5 extents:1 across:584708k SS
[ 5.451107] zram3: detected capacity change from 0 to 598745088
[ 5.473810] Adding 584708k swap on /dev/zram3. Priority:5 extents:1 across:584708k SS
[ 5.476850] zram4: detected capacity change from 0 to 598745088
[ 5.490975] Adding 584708k swap on /dev/zram4. Priority:5 extents:1 across:584708k SS
[ 5.498292] zram5: detected capacity change from 0 to 598745088
[ 5.510544] Adding 584708k swap on /dev/zram5. Priority:5 extents:1 across:584708k SS
[ 5.737532] gpio tegra-gpio wake18 for gpio=101(M:5)
[ 5.742398] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 6.813246] Software Queue-Pair Transport over NTB, version 4
[ 6.813594] tegra-pcie 10003000.pcie-controller: PCIE: Target abort, signature: 00201601
[ 6.822069] tegra-pcie 10003000.pcie-controller: PCIE: Target abort, signature: 00201705
[ 6.845975] switchtec 0000:01:00.1: NTB Transport QP 0 created
[ 6.846341] switchtec 0000:01:00.1: eth1 created
[ 6.976066] nvgpu: 17000000.gp10b railgate_enable_store:283 [INFO] railgate is disabled.
[ 6.988523] tegra-xusb 3530000.xhci: entering ELPG
[ 6.991388] tegra-xusb 3530000.xhci: entering ELPG done
[ 7.011644] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 8.065707] tegradc 15210000.nvdisplay: blank - powerdown
[ 8.740207] eqos 2490000.ether_qos eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 8.740940] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 8.777093] fuse init (API version 7.26)

Below is the output for the second TX2i.

root@tx2i-dev2:/home/nvidia# cat /sys/kernel/debug/bpmp/debug/emc/dram_info
Manufacturer ID (MR5): 6
Revision ID-1 (MR6): 4
Revision ID-2 (MR7): 0
IO Width/Density/Type (MR8): 0x08

root@tx2i-dev2:/home/nvidia# dmesg
[ 0.000000] Booting Linux on physical CPU 0x100
[ 0.000000] Linux version 4.9.140 (nvidia@tx2i-dev2) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1) ) #1 SMP PREEMPT Thu Nov 14 11:32:43 EST 2019
[ 0.000000] Boot CPU: AArch64 Processor [411fd073]
[ 0.000000] OF: fdt:memory scan node memory@80000000, reg size 16416,
[ 0.000000] OF: fdt: - 80000000 , 70000000
[ 0.000000] OF: fdt: - f0200000 , 145600000
[ 0.000000] OF: fdt: - 235e00000 , 200000
[ 0.000000] OF: fdt: - 236600000 , 200000
[ 0.000000] OF: fdt: - 237000000 , 200000
[ 0.000000] earlycon: uart8250 at MMIO32 0x0000000003100000 (options ‘’)
[ 0.000000] bootconsole [uart8250] enabled
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb0_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb0_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb1_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb1_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb2_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb2_carveout’: base 0x0000000000000000, size 0 MiB
[ 0.000000] OF: reserved mem: initialized node ramoops_carveout, compatible id nvidia,ramoops
[ 0.000000] cma: Reserved 64 MiB at 0x00000000fc000000
[ 0.000000] On node 0 totalpages: 1793024
[ 0.000000] DMA zone: 8192 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 523776 pages, LIFO batch:31
[ 0.000000] Normal zone: 19912 pages used for memmap
[ 0.000000] Normal zone: 1269248 pages, LIFO batch:31
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.0 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[

Please share the same information boot log and dram info for tx2.

Also tegrastats logs for both the machine after you run jetson_clocks

Hi,
When you connect two TX2s through a PCIe switch in NTB configuration, my understanding is that you would be accessing the memory of other TX2 (say TX2-B) through TX2-A’s BAR. right?
In which case, you can’t really expect the theoretical bandwidth here because this is not Tegra’s perf path. Tegra’s perf path is when a PCIe endpoint connected to one of TX2’s root ports and accessing TX2’s system memory.
In your case, what is happening is that, though it is incoming memory read/write traffic for TX2-B, for TX2, it is merely BAR access by CPU. The same is the case from the other side as well. So, this by design is not expected to give a good performance.
Although if the switch has a DMA engine and can push/pull data from either of the TX2s, it would be a perf path.
I hope this answers the perf being not up to the expectation part.
Coming to the disparity between the perf values in both paths (i.e. TX2i->TX2 being higher compared to TX2->TX2i). This is not expected provided memory frequency at which they both run is the same.
BTW, you mentioned that the link configuration is Gen-2 and x4 on TX2i and switch. Is it the same between TX2 and the switch also?
Also, please share ‘sudo lspci -vv’ on TX2 side.

I know this is old but just read it, i hope somebody is still interested.
From your response it sounds as if the BAR mechanism is to blame. I don’t think that is the case right? I mean, in PCIe all memory accesses happen thanks to the BAR mappings, whatever the agent performing the read/write the transaction is only possible if the target address is mapped into a device’s BAR.
I am new to the nvidia embedded world but I understand the problem here is the CPU is doing the accesses and not a DMA. CPUs access data over PCIe one 32bit word at a time, if the CPU is very high end and you are lucky perhaps a chache line at a time? not even sure if that is possible, but still, this will always be very slow compared to dedicated engines (such as DMA) specifically made to move data.

Hi boltzman,

Please open a new topic for your issue. Thanks