Tensorflow model memory issues with new docker version 19.01

Accelerated Computing NGC GPU Cloud Docker and NVIDIA Docker
1

Hello everyone,

I downloaded the recent tensorflow nvidia-docker release version 19.01. Previously, I was using 18.09 version in which my tensorflow model ran perfectly fine without any memory issues. But, when I run the same model on 19.01 version, it throws OOM (Out of memory) error. Just to make sure, there is no code change, I ran again in the 18.09 version and it doesn’t throw any error. Is such behavior expected in the new 19.01 version ?

2

I just want to add that I have a similar issue.

When running 19.01 my model uses 7900MB of RAM, but when I use a newer container/tensorflow version it jumps up over the max ram of the 2070 graphics card. I don’t really know if it’s a TensorFlow thing, or an Nvidia thing, but you are not alone with having this particular issue.