What is this error ? How bad is it ? Can it be ignored or needs to be fixed ? It is happening in the nvcr.io/nvidia/tensorflow:19.03-py2 container .
There should be no problems with correctness here. The error message is likely being triggered from tensorflow/arithmetic_optimizer.cc at r1.13 · tensorflow/tensorflow · GitHub
That check is inside a graph optimization step that simplifies slices of stack operations in certain cases (see https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/core/grappler/optimizers/arithmetic_optimizer.cc#L2940-L2957). The “error” means that the optimization will not get applied to the graph. In this case TF will simply execute the un-optimized (but functionally correct) graph.