Jetson TX2 inference time

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi, I’m using detection neural network with configuration of tiny yolo v2 and darknet’s framework.
I’m using the same code on both my computer, and the embbeded jetson TX2.
my computer’s GPU qualities:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core™ i7-7740X CPU @ 4.30GHz
Stepping: 9
CPU MHz: 800.741
CPU max MHz: 4500.0000
CPU min MHz: 800.0000
BogoMIPS: 8592.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti retpoline intel_pt rsb_ctxsw spec_ctrl tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

I’ve installed on both devicess cuda 9.0 and cuDNN 7.0.5.
On my host computer I got image detection within 0.0045 seconds (over 200 frames per second), and on the jetson TX2 I got image detection within 0.04 seconds, X10 slower (25 frames per second).
What makes the big difference? I know the host computer has 2 cores more then the jetson, and it has a bigger bandwidth of memory, but I don’t belive this is the main reason. so please, how can I get similar results on the jetson?
thank you all.

2

What’s your GPU on the desktop? The TX2 has 256 CUDA cores, but even a typical average PCIe desktop GPU might have thousands of cores.

Additional possibility: If the CUDA fits entirely in the GPU’s integrated memory on the PC, then this memory is far faster than the memory used on the Jetson (the Jetson GPU is integrated directly with the memory controller and does not have its own memory…what it has isn’t all that fast).

Finally, CPUs on a desktop PC are likely to have more/better cache. Don’t know, but I would expect cache to play part of this.

3

Yes, you right, of course it’s memory playing an important rool but I’ve tested the memory bandwidth of the desktop PC and got about 242 GB/s, against 39 GB/s on the jetson.
Although it’s a big difference, I should have the abillity to process an image of resulotion 448X448X3 fast enough… at list twice the speed I got now…

4

Someone else may have an idea, you probably can get better performance.

However, I suspect the PC bandwidth is actually more constrained on the Jetson than your bandwidth testing shows for the PC. I say this because on the desktop you are looking at what the PCIe bus limits to during measurement, but internally to the video card’s actual data throughput between GPU and its local RAM is far faster. It isn’t until you transfer back and forth over PCIe that this slows down. The RAM in the Jetson is much slower and always goes through the memory controller (shared by other parts of the o/s). In some cases the Jetson will use pinned memory such that cache is disabled (I couldn’t say when this is or is not the case, but how memory transfer is chosen quite often makes the difference in latency you want to improve).