Memory Alloc Error when using GPU computation

Hey Developers,

i am running Darkflow on my Jetson Tx2 (GitHub - thtrieu/darkflow: Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices)
and when i enable the ‘–gpu’ flag it gives me the following error:

... E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
... W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304

this is what the terminal gives me before he runs into the endless error (where i need to close the terminal to get out of it)

pciBusID: 0000:00:00.0
totalMemory: 7.67GiB freeMemory: 3.96GiB
2017-12-14 17:11:13.068942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)

How do i increase the freeMemory? Why is only 50% free? (i have nothing running but an Display).

Thanks!

Hi,

Sorry first that we don’t have experience on darkflow.

Buf from the log, it should be able to allocate more memory rather than just 3.96Gib.
Could you reboot the device and try it again.

More, could you also share the tegrastats information with us?

sudo ~/tegrastats

Thanks.

Is the executable built in 32-bit mode? If so, you may be running into an allocation limit for the process.

@AastaLLL the output is:

nvidia@tegra-ubuntu:~$ sudo ~/tegrastats 
RAM 1567/7851MB (lfb 1292x4MB) CPU [0%@2078,off,off,0%@2035,0%@2034,0%@2035] EMC_FREQ 1%@1600 GR3D_FREQ 20%@114 APE 150 BCPU@31C MCPU@31C GPU@37C PLL@31C AO@29.5C Tboard@27C Tdiode@29C PMIC@100C thermal@30.4C VDD_IN 1568/1568 VDD_CPU 229/229 VDD_GPU 153/153 VDD_SOC 382/382 VDD_WIFI 38/38 VDD_DDR 306/306

@snarky: I have no idea. Maybe you know if you look at its repository GitHub - thtrieu/darkflow: Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices

Hi,

We need to monitor the memory usage when running the darkflow.
Please keep running the tegrastats and execute darkflow at the same time to record the memory change.

Thanks.

Hi gustavvz,

Have you clarified the cause and resolved the problem?
Any result can be shared?

Thanks