fail to run tensorflow1.5 in tx2

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi,

Today I compile the tensorflow v1.5 in tx2.
I tried to run alexnet(npy), failed the following msg.

<dataset.ImageProducer object at 0x7f43bd7f90>
2018-02-07 10:29:46.851157: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:866] ARM has no NUMA node, hardcoding to return zero
2018-02-07 10:29:46.851333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.66GiB freeMemory: 5.91GiB
2018-02-07 10:29:46.851394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) → (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
start -----------------------------------------------------------------2018-02-07 10:29:50.875593
start------------run enqueue_paths_op------------
2018-02-07 10:29:51.241231: E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 1048576 bytes on host: CUDA_ERROR_UNKNOWN
2018-02-07 10:29:51.241341: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 1048576

Thanks.

2

Hi,

So far, we guess this error is caused by out of memory but are still checking it in detail.
To help us figure out the root cause, could you apply following experiments?

1. Run alexnet on TF CPU mode to check if an out of resource error.
2. Run MNIST model on GPU mode to check if an out of resource error.

Thanks.

3

Hi,

1，It’s OK to run alexnet on TF CPU mode.

2，It’s OK to run MNIST on TF GPU mode.

Thanks

4

Hi,

Guess that your error is caused by out of memory since MNIST is good on your environment.
Could you monitor TX2 memory usage via tegrastats and share it with us?

sudo ~/tegrastats

Thanks.