For multi-GPU, change --gpus based on your machine.
2022-06-09 09:42:36,674 [INFO] root: Registry: ['nvcr.io']
2022-06-09 09:42:36,780 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
2022-06-09 09:42:36,843 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/admin/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Loading experiment spec at %s. /workspace/tao-experiments/efficientdet/specs/efficientdet_d1_train.txt
Loading experiment spec at %s. /workspace/tao-experiments/efficientdet/specs/efficientdet_d1_train.txt
Using TensorFlow backend.
2022-06-09 09:42:43,724 [INFO] iva.efficientdet.utils.spec_loader: Merging specification from /workspace/tao-experiments/efficientdet/specs/efficientdet_d1_train.txt
Using TensorFlow backend.
2022-06-09 09:42:43,724 [INFO] iva.efficientdet.utils.spec_loader: Merging specification from /workspace/tao-experiments/efficientdet/specs/efficientdet_d1_train.txt
2022-06-09 09:42:43,726 [INFO] root: Horovod successfully initialized ...
2022-06-09 09:42:43,726 [INFO] root: [train] AMP is activated - Experiment Feature
2022-06-09 09:42:43,727 [INFO] root: Horovod successfully initialized ...
2022-06-09 09:42:43,727 [INFO] root: [train] AMP is activated - Experiment Feature
2022-06-09 09:42:43,727 [INFO] root: Starting training cycle: 1, epoch: 0.
2022-06-09 09:42:43,728 [INFO] root: Create EncryptCheckpointSaverHook.
2022-06-09 09:42:43,728 [INFO] root: Loading pretrained model...
2022-06-09 09:42:50,491 [INFO] iva.efficientdet.models.det_model_fn: LR schedule method: cosine
2022-06-09 09:42:50,846 [INFO] iva.efficientdet.models.det_model_fn: clip gradients norm by 5.000000
2022-06-09 09:43:00,293 [INFO] root: Starting training cycle: 1, epoch: 0.
2022-06-09 09:43:07,256 [INFO] iva.efficientdet.models.det_model_fn: LR schedule method: cosine
2022-06-09 09:43:07,614 [INFO] iva.efficientdet.models.det_model_fn: clip gradients norm by 5.000000
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [p6/kernel]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [p6/bias]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [p6_bn/gamma]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [p6_bn/beta]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [p6_bn/moving_mean]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [p6_bn/moving_variance]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_0/weighted_fusion_0_0]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_0/depthwise_kernel]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_0/pointwise_kernel]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_0/bias]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/gamma]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/beta]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/moving_mean]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/moving_variance]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0/kernel]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0/bias]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/gamma]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/beta]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/moving_mean]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/moving_variance]
2022-06-09 09:43:38,266 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_0/weighted_fusion_1_0]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_0/depthwise_kernel]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_0/pointwise_kernel]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_0/bias]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/gamma]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/beta]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/moving_mean]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/moving_variance]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0/kernel]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0/bias]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/gamma]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/beta]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/moving_mean]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/moving_variance]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_0/weighted_fusion_2_0]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_0/depthwise_kernel]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_0/pointwise_kernel]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_0/bias]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/gamma]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/beta]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/moving_mean]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/moving_variance]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0/kernel]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0/bias]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/gamma]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/beta]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/moving_mean]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/moving_variance]
2022-06-09 09:43:38,267 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_0/weighted_fusion_3_0]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_0/depthwise_kernel]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_0/pointwise_kernel]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_0/bias]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/gamma]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/beta]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/moving_mean]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/moving_variance]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0/kernel]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0/bias]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/gamma]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/beta]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/moving_mean]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/moving_variance]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_0/weighted_fusion_4_0]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_0/depthwise_kernel]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_0/pointwise_kernel]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_0/bias]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/gamma]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/beta]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/moving_mean]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/moving_variance]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0/kernel]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0/bias]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/gamma]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/beta]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/moving_mean]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/moving_variance]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_0/weighted_fusion_5_0]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_0/depthwise_kernel]
2022-06-09 09:43:38,268 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_0/pointwise_kernel]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_0/bias]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/gamma]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/beta]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/moving_mean]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/moving_variance]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_0/weighted_fusion_6_0]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_0/depthwise_kernel]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_0/pointwise_kernel]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_0/bias]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/gamma]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/beta]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/moving_mean]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/moving_variance]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_0/weighted_fusion_7_0]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_0/depthwise_kernel]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_0/pointwise_kernel]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_0/bias]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/gamma]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/beta]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/moving_mean]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/moving_variance]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_1/weighted_fusion_0_1]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_1/depthwise_kernel]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_1/pointwise_kernel]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_1/bias]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/gamma]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/beta]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/moving_mean]
2022-06-09 09:43:38,269 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/moving_variance]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_1/weighted_fusion_1_1]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_1/depthwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_1/pointwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_1/bias]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/gamma]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/beta]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/moving_mean]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/moving_variance]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_1/weighted_fusion_2_1]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_1/depthwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_1/pointwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_1/bias]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/gamma]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/beta]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/moving_mean]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/moving_variance]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_1/weighted_fusion_3_1]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_1/depthwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_1/pointwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_1/bias]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/gamma]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/beta]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/moving_mean]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/moving_variance]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_1/weighted_fusion_4_1]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_1/depthwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_1/pointwise_kernel]
2022-06-09 09:43:38,270 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_1/bias]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/gamma]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/beta]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/moving_mean]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/moving_variance]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_1/weighted_fusion_5_1]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_1/depthwise_kernel]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_1/pointwise_kernel]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_1/bias]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/gamma]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/beta]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/moving_mean]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/moving_variance]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_1/weighted_fusion_6_1]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_1/depthwise_kernel]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_1/pointwise_kernel]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_1/bias]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/gamma]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/beta]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/moving_mean]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/moving_variance]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_1/weighted_fusion_7_1]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_1/depthwise_kernel]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_1/pointwise_kernel]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_1/bias]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/gamma]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/beta]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/moving_mean]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/moving_variance]
2022-06-09 09:43:38,271 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_2/weighted_fusion_0_2]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_2/depthwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_2/pointwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_2/bias]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/gamma]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/beta]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/moving_mean]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/moving_variance]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_2/weighted_fusion_1_2]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_2/depthwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_2/pointwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_2/bias]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/gamma]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/beta]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/moving_mean]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/moving_variance]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_2/weighted_fusion_2_2]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_2/depthwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_2/pointwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_2/bias]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/gamma]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/beta]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/moving_mean]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/moving_variance]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_2/weighted_fusion_3_2]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_2/depthwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_2/pointwise_kernel]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_2/bias]
2022-06-09 09:43:38,272 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/gamma]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/beta]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/moving_mean]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/moving_variance]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_2/weighted_fusion_4_2]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_2/depthwise_kernel]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_2/pointwise_kernel]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_2/bias]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/gamma]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/beta]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/moving_mean]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/moving_variance]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_2/weighted_fusion_5_2]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_2/depthwise_kernel]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_2/pointwise_kernel]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_2/bias]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/gamma]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/beta]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/moving_mean]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/moving_variance]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_2/weighted_fusion_6_2]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_2/depthwise_kernel]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_2/pointwise_kernel]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_2/bias]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/gamma]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/beta]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/moving_mean]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/moving_variance]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_2/weighted_fusion_7_2]
2022-06-09 09:43:38,273 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_2/depthwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_2/pointwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_2/bias]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/gamma]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/beta]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/moving_mean]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/moving_variance]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_3/weighted_fusion_0_3]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_3/depthwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_3/pointwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_3/bias]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/gamma]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/beta]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/moving_mean]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/moving_variance]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_3/weighted_fusion_1_3]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_3/depthwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_3/pointwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_3/bias]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/gamma]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/beta]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/moving_mean]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/moving_variance]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_3/weighted_fusion_2_3]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_3/depthwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_3/pointwise_kernel]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_3/bias]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/gamma]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/beta]
2022-06-09 09:43:38,274 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/moving_mean]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/moving_variance]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_3/weighted_fusion_3_3]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_3/depthwise_kernel]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_3/pointwise_kernel]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_3/bias]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/gamma]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/beta]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/moving_mean]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/moving_variance]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_3/weighted_fusion_4_3]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_3/depthwise_kernel]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_3/pointwise_kernel]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_3/bias]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/gamma]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/beta]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/moving_mean]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/moving_variance]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_3/weighted_fusion_5_3]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_3/depthwise_kernel]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_3/pointwise_kernel]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_3/bias]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/gamma]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/beta]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/moving_mean]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/moving_variance]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_3/weighted_fusion_6_3]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_3/depthwise_kernel]
2022-06-09 09:43:38,275 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_3/pointwise_kernel]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_3/bias]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/gamma]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/beta]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/moving_mean]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/moving_variance]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_3/weighted_fusion_7_3]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_3/depthwise_kernel]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_3/pointwise_kernel]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_3/bias]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/gamma]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/beta]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/moving_mean]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/moving_variance]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_4/weighted_fusion_0_4]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_4/depthwise_kernel]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_4/pointwise_kernel]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_4/bias]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/gamma]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/beta]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/moving_mean]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/moving_variance]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_4/weighted_fusion_1_4]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_4/depthwise_kernel]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_4/pointwise_kernel]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_4/bias]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/gamma]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/beta]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/moving_mean]
2022-06-09 09:43:38,276 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/moving_variance]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_4/weighted_fusion_2_4]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_4/depthwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_4/pointwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_4/bias]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/gamma]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/beta]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/moving_mean]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/moving_variance]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_4/weighted_fusion_3_4]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_4/depthwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_4/pointwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_4/bias]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/gamma]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/beta]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/moving_mean]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/moving_variance]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_4/weighted_fusion_4_4]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_4/depthwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_4/pointwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_4/bias]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/gamma]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/beta]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/moving_mean]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/moving_variance]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_4/weighted_fusion_5_4]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_4/depthwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_4/pointwise_kernel]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_4/bias]
2022-06-09 09:43:38,277 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/gamma]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/beta]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/moving_mean]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/moving_variance]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_4/weighted_fusion_6_4]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_4/depthwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_4/pointwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_4/bias]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/gamma]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/beta]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/moving_mean]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/moving_variance]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_4/weighted_fusion_7_4]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_4/depthwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_4/pointwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_4/bias]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/gamma]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/beta]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/moving_mean]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/moving_variance]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-0/depthwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-0/pointwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-0/bias]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/gamma]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/beta]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/moving_mean]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/moving_variance]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-1/depthwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-1/pointwise_kernel]
2022-06-09 09:43:38,278 [WARNING] root: Checkpoint is missing variable [class-1/bias]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/gamma]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/beta]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/moving_mean]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/moving_variance]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2/depthwise_kernel]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2/pointwise_kernel]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2/bias]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/gamma]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/beta]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/moving_mean]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/moving_variance]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-predict/depthwise_kernel]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-predict/pointwise_kernel]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-predict/bias]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/gamma]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/beta]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/moving_mean]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/moving_variance]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/gamma]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/beta]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/moving_mean]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/moving_variance]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/gamma]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/beta]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/moving_mean]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/moving_variance]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/gamma]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/beta]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/moving_mean]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/moving_variance]
2022-06-09 09:43:38,279 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/moving_variance]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/moving_variance]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/moving_variance]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/moving_variance]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/moving_variance]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/moving_variance]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/moving_variance]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/gamma]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/beta]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/moving_mean]
2022-06-09 09:43:38,280 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/moving_variance]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0/depthwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0/pointwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0/bias]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/gamma]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/beta]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/moving_mean]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/moving_variance]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1/depthwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1/pointwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1/bias]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/gamma]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/beta]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/moving_mean]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/moving_variance]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-2/depthwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-2/pointwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-2/bias]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/gamma]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/beta]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/moving_mean]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/moving_variance]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-predict/depthwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-predict/pointwise_kernel]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-predict/bias]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/gamma]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/beta]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/moving_mean]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/moving_variance]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/gamma]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/beta]
2022-06-09 09:43:38,281 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/moving_mean]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/moving_variance]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/gamma]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/beta]
2022-06-09 09:43:38,282 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/moving_mean]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/moving_variance]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/gamma]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/beta]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/moving_mean]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/moving_variance]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/gamma]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/beta]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/moving_mean]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/moving_variance]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [current_loss_scale]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [good_steps]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_0/weighted_fusion_7_0/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [block5c_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [block3c_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [block6b_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [block6e_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_3/weighted_fusion_7_3/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [block2c_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,283 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block3b_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block6c_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block6d_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_4/weighted_fusion_3_4/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block3c_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block6b_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block6c_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block2a_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block3c_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_4/weighted_fusion_6_4/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block1b_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block2a_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [block3c_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,284 [WARNING] root: Checkpoint is missing variable [top_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [class-0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [class-1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [box-2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block4c_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block1b_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block2b_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block4a_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block4d_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_2/weighted_fusion_6_2/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block7a_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block5c_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block1b_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block1a_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block4a_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block4a_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block4c_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [block5a_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_1/weighted_fusion_2_1/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,285 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block2a_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block4d_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_4/weighted_fusion_7_4/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [box-predict/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block2b_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block2a_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block4b_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block4d_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block6d_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block3a_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block4a_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block4c_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block5c_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block2c_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block4b_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block5d_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block6b_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [block7a_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,286 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_1/weighted_fusion_5_1/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block3a_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block2c_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block4b_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block4b_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block5d_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block6a_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block7b_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_3/weighted_fusion_1_3/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block3a_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block3c_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block6a_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block6e_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block2b_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block3c_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,287 [WARNING] root: Checkpoint is missing variable [block3a_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block5b_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block7a_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block1a_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block5c_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block6c_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [top_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block6e_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block6b_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block7a_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block7b_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [class-predict/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block1a_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,288 [WARNING] root: Checkpoint is missing variable [block2a_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block6c_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [box-1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [box-2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block3c_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [p6_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block1b_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block3b_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block4a_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block4d_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block6e_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [p6/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block3c_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block4d_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block6e_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block7b_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block1b_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block1b_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block4a_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [block5a_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,289 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block2a_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block2c_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block4c_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block5a_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block7b_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_2/weighted_fusion_3_2/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block6b_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block4b_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block5b_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block6c_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block4b_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block5d_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_1/weighted_fusion_7_1/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block4d_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block6d_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block6d_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block2c_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block2c_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,290 [WARNING] root: Checkpoint is missing variable [block4b_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block5d_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block6b_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block7a_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [top_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block6e_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block2c_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block4b_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block5d_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block5d_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block6a_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block6e_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [class-0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block2c_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block6b_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block3a_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block4a_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block5b_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [block6a_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,291 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block4d_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block5d_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block5c_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block5b_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block3a_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block5b_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block6d_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block6e_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block7a_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_0/weighted_fusion_6_0/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block2b_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block6a_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block1b_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block2b_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block3b_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block6d_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block7b_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [block3c_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,292 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block4b_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block5c_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block6c_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block5b_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block3b_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block1a_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block2a_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block2a_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block2c_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block3b_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block3c_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block5a_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block5a_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_1/weighted_fusion_1_1/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block4b_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block5d_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block1b_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block2a_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block3b_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,293 [WARNING] root: Checkpoint is missing variable [block4a_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block6d_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block7b_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_3/weighted_fusion_4_3/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block6a_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block7a_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block2b_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block4c_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block5a_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block5c_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block5d_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_3/weighted_fusion_0_3/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block2b_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block4b_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,294 [WARNING] root: Checkpoint is missing variable [block5a_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block5d_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block6c_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block6e_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block4a_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block6c_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block2c_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block4a_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block5b_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block5b_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block6a_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block7a_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_0/weighted_fusion_1_0/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block2c_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block5c_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [class-0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block2c_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block4c_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [block5d_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,295 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block5d_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block3a_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block4d_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block5b_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block6a_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block6c_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [box-predict/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block3a_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block3a_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block3c_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block5b_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block6a_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block6d_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [block6e_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,296 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_4/weighted_fusion_4_4/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [stem_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block6b_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block6c_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block6d_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_3/weighted_fusion_6_3/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block1b_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block7b_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block6a_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [class-2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block3b_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [p6/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block4d_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block6d_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block7b_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [box-2-bn-5/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block3c_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,297 [WARNING] root: Checkpoint is missing variable [block5a_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block2a_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block3b_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block5a_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block6b_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block7b_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block2a_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block6b_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block6c_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [class-2-bn-7/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block2a_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block3b_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block3b_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block4d_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block6b_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_2/weighted_fusion_2_2/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [class-2-bn-6/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block2b_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block2c_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,298 [WARNING] root: Checkpoint is missing variable [block5a_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_2/weighted_fusion_5_2/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block6e_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block4c_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block5a_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block5d_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block6e_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_4/weighted_fusion_1_4/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block1b_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block4a_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block4b_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block4c_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block5b_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block6c_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [block7a_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,299 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_0/weighted_fusion_2_0/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [class-1-bn-4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block6d_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block4c_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block4c_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block5c_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [box-0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [box-1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [class-1-bn-5/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block2b_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block3a_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block5b_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [block6a_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [p6_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_1/weighted_fusion_4_1/ExponentialMovingAverage]
2022-06-09 09:43:38,300 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [class-2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [box-2-bn-7/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block1a_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block3a_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block5b_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block6a_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block6d_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block3a_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block5d_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block6c_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [box-1-bn-7/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [stem_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block4a_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block6b_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block6c_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [top_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block6d_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block7a_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [block5a_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,301 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block2a_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block3b_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_1/weighted_fusion_0_1/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block7b_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_4/weighted_fusion_2_4/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block1a_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block4d_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block7b_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block2a_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block3b_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block3c_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block5a_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block6b_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block7a_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block7b_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_3/weighted_fusion_3_3/ExponentialMovingAverage]
2022-06-09 09:43:38,302 [WARNING] root: Checkpoint is missing variable [block4a_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block4a_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block3b_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block6b_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [weighted_fusion_7_2/weighted_fusion_7_2/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block2a_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block3b_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_2/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [class-0-bn-7/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block3b_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block3a_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block4b_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_0/weighted_fusion_0_0/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_0/weighted_fusion_3_0/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [box-predict/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block2b_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block3b_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block4c_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [block6a_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,303 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block2c_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block4c_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block5a_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block6e_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block7a_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block7b_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block4c_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block4d_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block5b_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block6a_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block6c_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block4b_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block7b_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block4c_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block5c_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,304 [WARNING] root: Checkpoint is missing variable [block7a_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [weighted_fusion_6_1/weighted_fusion_6_1/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block4c_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [class-2-bn-3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block1a_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block2b_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block5c_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [p6_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [class-2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [box-2-bn-3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block3a_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block1a_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block1a_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block2b_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block3a_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block3b_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block6d_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block6e_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [block7a_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,305 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block7a_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block5d_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [stem_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block1b_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block2c_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block6e_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [top_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [weighted_fusion_1_2/weighted_fusion_1_2/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [stem_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block6b_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block5b_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block5d_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block7b_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block2a_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_0/weighted_fusion_5_0/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [box-0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block3a_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block3b_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,306 [WARNING] root: Checkpoint is missing variable [block3c_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block4a_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block6b_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_4/weighted_fusion_0_4/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block1b_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block1b_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block3c_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block6d_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block6e_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block7b_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block6b_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block3a_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [block5d_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_0/weighted_fusion_4_0/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,307 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_7_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block2a_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block2c_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block4d_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block6a_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [class-2-bn-5/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [bifpn0_0_8_0_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [box-2-bn-4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block2c_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block4a_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block4c_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block6a_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block7a_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [box-0-bn-7/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block2c_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block6a_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block6e_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [bifpn0_1_9_0_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [class-0-bn-3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block4d_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [block5c_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,308 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [box-2/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block4c_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block5d_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block6c_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block5b_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [p6_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [box-0-bn-5/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block4a_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block4b_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block4d_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [weighted_fusion_3_1/weighted_fusion_3_1/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_3/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [class-predict/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [class-2-bn-4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [box-0-bn-4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block6d_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block5a_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block5c_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block1a_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block1a_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block2b_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [block5c_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,309 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block4c_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block5a_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block2b_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [box-1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block4b_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block5c_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block6b_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block1a_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block2b_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block2b_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block5c_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block6e_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block7a_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block7b_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [class-0-bn-5/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [class-1-bn-6/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [class-1-bn-7/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block6c_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block6d_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [block6e_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_0/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,310 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_3/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_3/weighted_fusion_5_3/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block4b_project_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_0/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_4/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block2a_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block3c_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block5b_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [weighted_fusion_2_3/weighted_fusion_2_3/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [box-0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [box-1-bn-4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block2a_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block5a_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [box-0-bn-6/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block3c_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [block6b_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_0/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_3/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_4/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [class-1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,311 [WARNING] root: Checkpoint is missing variable [box-0-bn-3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn0_2_10_0_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_3/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block6d_expand_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block3c_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block3c_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block4a_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block5a_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block6c_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block7b_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_1/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_1/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_2/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_2_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_1/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block4a_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block6c_dwconv/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_0_4/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_4/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [block3a_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [box-1-bn-5/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,312 [WARNING] root: Checkpoint is missing variable [class-0-bn-4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block7a_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [bifpn_bn_5_4/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block1b_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block4d_expand_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block7a_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_2/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_6_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [class-1-bn-3/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [box-2-bn-6/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block5b_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block6a_expand_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_5_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block3a_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block4d_project_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block5c_expand_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block6d_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_3_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [weighted_fusion_4_2/weighted_fusion_4_2/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_4/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block2b_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [block3b_project_bn/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [bifpn_bn_1_2/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,313 [WARNING] root: Checkpoint is missing variable [stem_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block4b_se_reduce/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block4d_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block4d_se_expand/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block5b_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block6a_project_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_4_1/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_2_2/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [bifpn_bn_4_2/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [class-predict/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block5a_bn/moving_mean/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block5c_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [bifpn_bn_3_0/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_0_0/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block1a_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block2b_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block2c_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block4b_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [weighted_fusion_5_4/weighted_fusion_5_4/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [bifpn_bn_6_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [box-1-bn-6/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [bifpn_bn_7_1/moving_variance/ExponentialMovingAverage]
2022-06-09 09:43:38,314 [WARNING] root: Checkpoint is missing variable [block1a_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [block2b_se_expand/bias/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [block4c_expand_conv/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [block5c_project_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [block5d_se_reduce/kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [block7b_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [bifpn0_2_6_0_bn/beta/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [bifpn0_1_7_0_bn/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [weighted_fusion_0_2/weighted_fusion_0_2/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [after_combine_dw_conv_1_4/pointwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [class-1/depthwise_kernel/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [class-0-bn-6/gamma/ExponentialMovingAverage]
2022-06-09 09:43:38,315 [WARNING] root: Checkpoint is missing variable [box-1-bn-3/moving_variance/ExponentialMovingAverage]
2022-06-09 09:44:42,293 [INFO] root: Saving checkpoints for 0 into /workspace/tao-experiments/efficientdet/experiment_dir_unpruned/model.step-0.tlt.
[GPU 00] Restoring pretrained weights (437 Tensors)
2022-06-09 09:44:46,212 [INFO] root: Pretrained weights loaded with success...

82d25dba0fb1:108:240 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
82d25dba0fb1:108:240 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
82d25dba0fb1:108:240 [0] NCCL INFO NET/IB : No device found.
82d25dba0fb1:108:240 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.71<0>
82d25dba0fb1:108:240 [0] NCCL INFO Using network Socket
NCCL version 2.9.9+cuda11.3
82d25dba0fb1:109:243 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0>
82d25dba0fb1:109:243 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
82d25dba0fb1:109:243 [0] NCCL INFO NET/IB : No device found.
82d25dba0fb1:109:243 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.71<0>
82d25dba0fb1:109:243 [0] NCCL INFO Using network Socket
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC


82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01/02 :    0   1
82d25dba0fb1:108:240 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
82d25dba0fb1:109:243 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
82d25dba0fb1:108:240 [0] NCCL INFO Channel 00 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 00 : 1[c1000] -> 0[81000] via P2P/IPC
82d25dba0fb1:108:240 [0] NCCL INFO Channel 01 : 0[81000] -> 1[c1000] via P2P/IPC
82d25dba0fb1:109:243 [0] NCCL INFO Channel 01 : 1[c1000] -> 0[81000] via P2P/IPC

82d25dba0fb1:108:240 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:108:240 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:742 -> 1

82d25dba0fb1:109:243 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'peer access is not supported between these two devices'
82d25dba0fb1:109:243 [0] NCCL INFO transport/p2p.cc:238 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO transport.cc:111 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:742 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:867 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:108:240 [0] NCCL INFO init.cc:916 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:903 -> 1
82d25dba0fb1:109:243 [0] NCCL INFO init.cc:916 -> 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: ncclCommInitRank failed: unhandled cuda error
	 [[{{node DistributedMomentumOptimizer_Allreduce/cond_296/HorovodAllreduce_DistributedMomentumOptimizer_Allreduce_cond_296_Cast_1_0}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 117, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 63, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 71, in run_executer
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/executer/distributed_executer.py", line 335, in train_and_eval
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: ncclCommInitRank failed: unhandled cuda error
	 [[node DistributedMomentumOptimizer_Allreduce/cond_296/HorovodAllreduce_DistributedMomentumOptimizer_Allreduce_cond_296_Cast_1_0 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'DistributedMomentumOptimizer_Allreduce/cond_296/HorovodAllreduce_DistributedMomentumOptimizer_Allreduce_cond_296_Cast_1_0':
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 117, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 63, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 71, in run_executer
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/executer/distributed_executer.py", line 335, in train_and_eval
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1191, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/models/det_model_fn.py", line 657, in efficientdet_model_fn
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/models/det_model_fn.py", line 547, in _model_fn
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/experimental/loss_scale_optimizer.py", line 123, in compute_gradients
    grad_loss=grad_loss)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 475, in compute_gradients
    avg_grads = self._allreduce_grads(grads, vars)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 398, in allreduce_grads
    for grad in grads]
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 398, in <listcomp>
    for grad in grads]
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 241, in _allreduce_cond
    allreduce_fn, id_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1224, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1061, in BuildCondBranch
    original_result = fn()
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 235, in allreduce_fn
    return allreduce(tensor, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 123, in allreduce
    name=name)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/mpi_ops.py", line 121, in _allreduce
    ignore_name_scope=ignore_name_scope)
  File "<string>", line 102, in horovod_allreduce
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: ncclCommInitRank failed: unhandled cuda error
	 [[{{node DistributedMomentumOptimizer_Allreduce/cond_296/HorovodAllreduce_DistributedMomentumOptimizer_Allreduce_cond_296_Cast_1_0}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 117, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 63, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 71, in run_executer
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/executer/distributed_executer.py", line 335, in train_and_eval
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: ncclCommInitRank failed: unhandled cuda error
	 [[node DistributedMomentumOptimizer_Allreduce/cond_296/HorovodAllreduce_DistributedMomentumOptimizer_Allreduce_cond_296_Cast_1_0 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'DistributedMomentumOptimizer_Allreduce/cond_296/HorovodAllreduce_DistributedMomentumOptimizer_Allreduce_cond_296_Cast_1_0':
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 117, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 63, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/scripts/train.py", line 71, in run_executer
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/executer/distributed_executer.py", line 335, in train_and_eval
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1191, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/models/det_model_fn.py", line 657, in efficientdet_model_fn
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/efficientdet/models/det_model_fn.py", line 547, in _model_fn
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/experimental/loss_scale_optimizer.py", line 123, in compute_gradients
    grad_loss=grad_loss)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 475, in compute_gradients
    avg_grads = self._allreduce_grads(grads, vars)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 398, in allreduce_grads
    for grad in grads]
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 398, in <listcomp>
    for grad in grads]
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 241, in _allreduce_cond
    allreduce_fn, id_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1224, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1061, in BuildCondBranch
    original_result = fn()
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 235, in allreduce_fn
    return allreduce(tensor, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 123, in allreduce
    name=name)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/mpi_ops.py", line 121, in _allreduce
    ignore_name_scope=ignore_name_scope)
  File "<string>", line 102, in horovod_allreduce
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun.real detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[3248,1],1]
  Exit code:    1
--------------------------------------------------------------------------
2022-06-09 09:46:18,714 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.