Full processing only in one GPU, how to leverage the processing to another GPU?


When we put our code into the tensorflow docker image and run it, only one GPU is used. However on this GPU the Memory usage is near 100% (31344MiB/32475MiB) , but the GPU usage is only around 20-30%. When we run the CNN with the defect images the Memory is full to and the GPU usage is at 70-80% (also only one GPU used).

The question here is how to also leverage the running to the other GPU so that the load for GPU-volatile and Memory could be balance out. ?


Hi Christina,

By default the TensorFlow memory allocator claims all device memory for itself. Internally, tensorflow allocations are then made out of that memory pool. This has advantages for avoiding memory fragmentation, but makes sharing a device between multiple processes impossible.

There are two mechanisms available to reduce the amount of device memory TensorFlow uses. Both are controlled by session config options.

Setting config.gpu_options.allow_growth = True forces TensorFlow to allocate memory as it is required.

Alternatively, setting config.gpu_options.per_process_gpu_memory_fraction = 0.45 causes TF to claim only 45% of the devices total memory. This guarantees memory will remain available for another process, but models requiring more than 45% of the memory will terminate with out-of-memory errors.

For further details see https://www.tensorflow.org/guide/using_gpu#allowing_gpu_memory_growth