attempting to restore from ./network_checkpoint_kirsh_4/ 2021-06-14 16:45:24.123809: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Invalid argument: tensor_name = global_step; expected dtype int64 does not equal original dtype int32 Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: tensor_name = global_step; expected dtype int64 does not equal original dtype int32 [[{{node save/RestoreV2}}]] [[save/RestoreV2/_15]] (1) Invalid argument: tensor_name = global_step; expected dtype int64 does not equal original dtype int32 [[{{node save/RestoreV2}}]] 0 successful operations. 0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 1290, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: tensor_name = global_step; expected dtype int64 does not equal original dtype int32 [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[save/RestoreV2/_15]] (1) Invalid argument: tensor_name = global_step; expected dtype int64 does not equal original dtype int32 [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] 0 successful operations. 0 derived errors ignored. Original stack trace for 'save/RestoreV2': File "post_process.py", line 215, in ctr.run() File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/controller.py", line 93, in run self.solver.eval() File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/solver.py", line 682, in eval self.load_network(domain_index, iteration_index) File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/solver.py", line 262, in load_network saver = tf.train.Saver(tf.global_variables(), max_to_keep=1) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 828, in __init__ self.build() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 840, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 878, in _build build_restore=build_restore) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps restore_sequentially) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2 name=name) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "post_process.py", line 215, in ctr.run() File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/controller.py", line 93, in run self.solver.eval() File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/solver.py", line 682, in eval self.load_network(domain_index, iteration_index) File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/solver.py", line 266, in load_network saver.restore(self.sess, ckpt.model_checkpoint_path) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 1326, in restore err, "a mismatch between the current graph and the graph") tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: 2 root error(s) found. (0) Invalid argument: tensor_name = global_step; expected dtype int64 does not equal original dtype int32 [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[save/RestoreV2/_15]] (1) Invalid argument: tensor_name = global_step; expected dtype int64 does not equal original dtype int32 [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] 0 successful operations. 0 derived errors ignored. Original stack trace for 'save/RestoreV2': File "post_process.py", line 215, in ctr.run() File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/controller.py", line 93, in run self.solver.eval() File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/solver.py", line 682, in eval self.load_network(domain_index, iteration_index) File "/usr/local/lib/python3.7/dist-packages/simnet-21.6-py3.7.egg/simnet/solver.py", line 262, in load_network saver = tf.train.Saver(tf.global_variables(), max_to_keep=1) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 828, in __init__ self.build() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 840, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 878, in _build build_restore=build_restore) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps restore_sequentially) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2 name=name) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__ self._traceback = tf_stack.extract_stack() [b61721caacdb:11314] *** Process received signal *** [b61721caacdb:11314] Signal: Segmentation fault (11) [b61721caacdb:11314] Signal code: Address not mapped (1) [b61721caacdb:11314] Failing at address: 0x7f32860a920d [b61721caacdb:11314] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f328935d980] [b61721caacdb:11314] [ 1] /lib/x86_64-linux-gnu/libc.so.6(getenv+0xa5)[0x7f3288f9c8a5] [b61721caacdb:11314] [ 2] /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4(_ZN13TCMallocGuardD1Ev+0x34)[0x7f3289807e44] [b61721caacdb:11314] [ 3] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xf5)[0x7f3288f9d735] [b61721caacdb:11314] [ 4] /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4(+0x13cb3)[0x7f3289805cb3] [b61721caacdb:11314] *** End of error message ***