Hello everyone,
I pulled the recent release of tensorflow nvidia-docker version 19.01 and faced the following error while running models:
"
2019-01-28 20:31:37.275872: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:33:19.926346: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:35:37.982278: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:35:51.655399: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:52:38.266663: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:54:24.258871: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:56:41.890462: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 0, topological sort failed with message: The graph couldn’t be sorted in topological order.
2019-01-28 20:56:55.737031: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:666] Iteration = 1, topological sort failed with message: The graph couldn’t be sorted in topological order.
"
This error shows up at the start of training and every sometimes between epochs. The previous docker versions didn’t throw this error (for the same codes/models). Did anyone face this issue ? Please help me out with this.