run tensorflow on Jetson TX2 with GPU out wrong result

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

On TX2, I have installed Jetpack3.3 and Tensorflow 1.9.0 use:
pip3 install --extra-index-url=https://developer.download.nvidia.com/compute/redist/jp33 tensorflow-gpu

run graph on CPU make right result but run on GPU make wrong result and out different results with same input
example code: https://1drv.ms/u/s!AhFk3ICqlZI2irJdcWsWHKtBHhrt5w

run on CPU: CUDA_VISIBLE_DEVICES=‘’ python3 test.py
run on GPU: python3 test.py

what error?

2

Hi,

Do you meet any error when executing?
Or the application can finish successfully but with the incorrect results?

We will check your sample and update more information with you later.
Thanks

3

no error when executing
only have notify: ARM64 does not support NUMA - returning NUMA node zero
the application finish successfully with the incorrect results

I tried build tensorflow C++ on TX2 follow: GitHub - JasonAtNvidia/JetsonTFBuild: Assistance script to build TensorFlow on an NVIDIA Jetson Module, but same error

this is text file define graph: https://1drv.ms/u/s!AhFk3ICqlZI2irJeV3K_0KyvMB8mcw

4

Exactly the same problem occurred in my environment. (TX2, Jetpack3.3, Tensorflow 1.9.0, my task is object ditection with SSD, too).
Further, I tryed to run the same program on a windows PC with GeForce, I got right result.

5

Hi,

This is a known issue. TensorFlow may introduce incorrect result with SpaceToBatchND.
You can check this topic for information:
https://devtalk.nvidia.com/default/topic/1037898

This issue is fixed in the Xavier platform.
We test your sample on Xavier and can get the identical result with CPU and GPU.

CPU

[[[ 74.377      320.73145    139.79294    377.59125      0.99432826
     1.        ]
  [  1.4652786  299.86053     71.74264    367.73016      0.9911488
     1.        ]
  [119.930405     0.9353714  265.55936    113.78609      0.96522236
     1.        ]
  [ 67.6715     183.38559    200.57294    264.9446       0.9565499
     1.        ]
  [187.84546    174.78589    271.09146    259.42136      0.9555303
     1.        ]
  [  3.7699127  210.44226     66.127335   268.4802       0.94328606
     1.        ]]]

GPU

[[[ 74.37714    320.73148    139.79294    377.59125      0.99432814
     1.        ]
  [  1.4652481  299.86063     71.74257    367.7301       0.9911486
     1.        ]
  [119.930435     0.9353714  265.5593     113.786095     0.9652203
     1.        ]
  [ 67.67179    183.38574    200.57285    264.94446      0.95654756
     1.        ]
  [187.84558    174.78603    271.0915     259.4213       0.9555299
     1.        ]
  [  3.7699547  210.44241     66.12728    268.48013      0.94328594
     1.        ]]]

Thanks.