Tensorflow topological sort failed with new docker version 19.01

Hello everyone,

I pulled the recent release of tensorflow nvidia-docker version 19.01 and faced the following error while running models:

"
2019-01-28 20:31:37.275872: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:33:19.926346: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:35:37.982278: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:35:51.655399: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:52:38.266663: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:54:24.258871: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:56:41.890462: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:56:55.737031: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.

"

This error shows up at the start of training and every sometimes between epochs. The previous docker versions didn’t throw this error (for the same codes/models). Did anyone face this issue ? Please help me out with this.

I also face this issue. Any news on that?

Not yet, I am just using older versions for now. Did you also face any OOM issue ?

No, just the same warning that you also get.

Ohh Okay. I observed there is some difference in the memory too between the older and newer versions. In older versions I was able to use higher batch sizes and couldn’t do that in newer versions.