Could not allocate memory: Tensorflow 1.5 on python 3 for Jetson TX2

Hey guys,

I found this forum to be incredibly helpful to my projects and saved me a lot of headache. I was able to get Tensorflow 1.5 installed on my TX2 compile for python3, but operations fail horribly…

My build configuration scripts are here:
https://github.com/frankgu968/installTensorFlowTX2

I am running the CUDA 9.0 + cuDNN 7.0 with tensorflow 1.5 compiled for python 3 on the JetPack 3.2 DP. Any basic operation currently fails…

When I start the tf session, it prompts me that “tensorflow/core/common_runtime/gpu/gpu_device.cc:859] Could not identify NUMA node of /job:localhost/replica:0/task:0/device:GPU:0, defaulting to 0. Your kernel may not have been built with NUMA support.”

Running any basic session fails with the message:
“tensorflow/stream_executor/cuda/cuda_driver.cc: 967] failed to alloc 2308 bytes on host: CUDA_ERROR_UNKOWN”
"tensorflow/stream_executor/common_runtime/gpu/pool_allocator.h: 195] could not allocate pinned host memory of size: 2308 "

Any ideas on what the problem is or how to fix it?

Hi,

We have built TensorFlow-1.5 on JetPack3.2 DP internally, it can work correctly.

1. This NUMA log is a harmless warning. TensorFlow still can run successfully with it.

2. This is a memory related error. To check if it’s caused by OOM, could you run TF with MNIST sample first?

Thanks.

Hi Aastall,

How about sharing your wheel ?

Best regards,
Kalevi

Hi Aastall,

Never mind. I saw you provided a link to a wheel in another thread.

https://devtalk.nvidia.com/default/topic/1029572/jetson-tx2/could-not-allocate-memory-tensorflow-1-5-on-python-3-for-jetson-tx2-/?offset=3#5239284

Best regards,
Kalevi