@Morganh, here is the log that I get when I start the training let’s say with batch size 4 and then stop the train process and restart the training process with an increased batch size 8.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: assertion failed: [2.66666675]
[[{{node Assert/AssertGuard/Assert}}]]
[[resnet18_nopool_bn_detectnet_v2/block_4b_bn_2/AssignMovingAvg/_4217]]
(1) Invalid argument: assertion failed: [2.66666675]
[[{{node Assert/AssertGuard/Assert}}]]
0 successful operations.
0 derived errors ignored.
The moment I revert the batch size the training continues from where it was left off.