2022-05-24 13:17:24,792 [INFO] root: Registry: ['nvcr.io']
2022-05-24 13:17:24,869 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
2022-05-24 13:17:24,965 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/ataka/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
/usr/local/lib/python3.6/dist-packages/numba/cuda/envvars.py:17: NumbaWarning: 
Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib64/libnvvm.so.

For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup')
  warnings.warn(errors.NumbaWarning(msg))
/usr/local/lib/python3.6/dist-packages/numba/cuda/envvars.py:17: NumbaWarning: 
Environment variables with the 'NUMBAPRO' prefix are deprecated and consequently ignored, found use of NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice/.

For more information about alternatives visit: ('http://numba.pydata.org/numba-doc/latest/cuda/overview.html', '#cudatoolkit-lookup')
  warnings.warn(errors.NumbaWarning(msg))
2022-05-24 04:17:32,048 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/unet/specs/unet_train_resnet_unet_kitti_retrain.txt
2022-05-24 04:17:32,055 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0
2022-05-24 04:17:32,055 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1
2022-05-24 04:17:32,055 [INFO] iva.unet.model.utilities: Label Id 2: Train Id 2
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 3: Train Id 3
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 4: Train Id 4
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 5: Train Id 5
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 6: Train Id 6
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 7: Train Id 7
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 8: Train Id 8
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 9: Train Id 9
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 10: Train Id 10
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 11: Train Id 11
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 12: Train Id 12
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 13: Train Id 13
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 14: Train Id 14
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 15: Train Id 15
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 16: Train Id 16
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 17: Train Id 17
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 18: Train Id 18
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 19: Train Id 19
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 20: Train Id 20
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 21: Train Id 21
2022-05-24 04:17:32,056 [INFO] iva.unet.model.utilities: Label Id 22: Train Id 22
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 23: Train Id 23
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 24: Train Id 24
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 25: Train Id 25
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 26: Train Id 26
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 27: Train Id 27
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 28: Train Id 28
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 29: Train Id 29
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 30: Train Id 30
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 31: Train Id 31
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 32: Train Id 32
2022-05-24 04:17:32,057 [INFO] iva.unet.model.utilities: Label Id 33: Train Id 33
2022-05-24 04:17:32,057 [INFO] iva.unet.model.model_io: Loading weights from /workspace/tao-experiments/unet/kitti_experiment_retrain/weights/model_kitti_retrained.tlt
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 3, 320, 320)  0                                            
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 64, 160, 160) 9472        input_1[0][0]                    
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 64, 160, 160) 0           conv1[0][0]                      
__________________________________________________________________________________________________
block_1a_conv_1 (Conv2D)        (None, 64, 80, 80)   36928       activation_1[0][0]               
__________________________________________________________________________________________________
block_1a_relu_1 (Activation)    (None, 64, 80, 80)   0           block_1a_conv_1[0][0]            
__________________________________________________________________________________________________
block_1a_conv_2 (Conv2D)        (None, 64, 80, 80)   36928       block_1a_relu_1[0][0]            
__________________________________________________________________________________________________
block_1a_conv_shortcut (Conv2D) (None, 64, 80, 80)   4160        activation_1[0][0]               
__________________________________________________________________________________________________
add_1 (Add)                     (None, 64, 80, 80)   0           block_1a_conv_2[0][0]            
                                                                 block_1a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_1a_relu (Activation)      (None, 64, 80, 80)   0           add_1[0][0]                      
__________________________________________________________________________________________________
block_1b_conv_1 (Conv2D)        (None, 64, 80, 80)   36928       block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_relu_1 (Activation)    (None, 64, 80, 80)   0           block_1b_conv_1[0][0]            
__________________________________________________________________________________________________
block_1b_conv_2 (Conv2D)        (None, 64, 80, 80)   36928       block_1b_relu_1[0][0]            
__________________________________________________________________________________________________
block_1b_conv_shortcut (Conv2D) (None, 64, 80, 80)   4160        block_1a_relu[0][0]              
__________________________________________________________________________________________________
add_2 (Add)                     (None, 64, 80, 80)   0           block_1b_conv_2[0][0]            
                                                                 block_1b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_1b_relu (Activation)      (None, 64, 80, 80)   0           add_2[0][0]                      
__________________________________________________________________________________________________
block_2a_conv_1 (Conv2D)        (None, 120, 40, 40)  69240       block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_2a_relu_1 (Activation)    (None, 120, 40, 40)  0           block_2a_conv_1[0][0]            
__________________________________________________________________________________________________
block_2a_conv_2 (Conv2D)        (None, 128, 40, 40)  138368      block_2a_relu_1[0][0]            
__________________________________________________________________________________________________
block_2a_conv_shortcut (Conv2D) (None, 128, 40, 40)  8320        block_1b_relu[0][0]              
__________________________________________________________________________________________________
add_3 (Add)                     (None, 128, 40, 40)  0           block_2a_conv_2[0][0]            
                                                                 block_2a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_2a_relu (Activation)      (None, 128, 40, 40)  0           add_3[0][0]                      
__________________________________________________________________________________________________
block_2b_conv_1 (Conv2D)        (None, 120, 40, 40)  138360      block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_relu_1 (Activation)    (None, 120, 40, 40)  0           block_2b_conv_1[0][0]            
__________________________________________________________________________________________________
block_2b_conv_2 (Conv2D)        (None, 112, 40, 40)  121072      block_2b_relu_1[0][0]            
__________________________________________________________________________________________________
block_2b_conv_shortcut (Conv2D) (None, 112, 40, 40)  14448       block_2a_relu[0][0]              
__________________________________________________________________________________________________
add_4 (Add)                     (None, 112, 40, 40)  0           block_2b_conv_2[0][0]            
                                                                 block_2b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_2b_relu (Activation)      (None, 112, 40, 40)  0           add_4[0][0]                      
__________________________________________________________________________________________________
block_3a_conv_1 (Conv2D)        (None, 208, 20, 20)  209872      block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_3a_relu_1 (Activation)    (None, 208, 20, 20)  0           block_3a_conv_1[0][0]            
__________________________________________________________________________________________________
block_3a_conv_2 (Conv2D)        (None, 224, 20, 20)  419552      block_3a_relu_1[0][0]            
__________________________________________________________________________________________________
block_3a_conv_shortcut (Conv2D) (None, 224, 20, 20)  25312       block_2b_relu[0][0]              
__________________________________________________________________________________________________
add_5 (Add)                     (None, 224, 20, 20)  0           block_3a_conv_2[0][0]            
                                                                 block_3a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_3a_relu (Activation)      (None, 224, 20, 20)  0           add_5[0][0]                      
__________________________________________________________________________________________________
block_3b_conv_1 (Conv2D)        (None, 216, 20, 20)  435672      block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_relu_1 (Activation)    (None, 216, 20, 20)  0           block_3b_conv_1[0][0]            
__________________________________________________________________________________________________
block_3b_conv_2 (Conv2D)        (None, 216, 20, 20)  420120      block_3b_relu_1[0][0]            
__________________________________________________________________________________________________
block_3b_conv_shortcut (Conv2D) (None, 216, 20, 20)  48600       block_3a_relu[0][0]              
__________________________________________________________________________________________________
add_6 (Add)                     (None, 216, 20, 20)  0           block_3b_conv_2[0][0]            
                                                                 block_3b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_3b_relu (Activation)      (None, 216, 20, 20)  0           add_6[0][0]                      
__________________________________________________________________________________________________
block_4a_conv_1 (Conv2D)        (None, 416, 20, 20)  809120      block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_4a_relu_1 (Activation)    (None, 416, 20, 20)  0           block_4a_conv_1[0][0]            
__________________________________________________________________________________________________
block_4a_conv_2 (Conv2D)        (None, 392, 20, 20)  1468040     block_4a_relu_1[0][0]            
__________________________________________________________________________________________________
block_4a_conv_shortcut (Conv2D) (None, 392, 20, 20)  85064       block_3b_relu[0][0]              
__________________________________________________________________________________________________
add_7 (Add)                     (None, 392, 20, 20)  0           block_4a_conv_2[0][0]            
                                                                 block_4a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_4a_relu (Activation)      (None, 392, 20, 20)  0           add_7[0][0]                      
__________________________________________________________________________________________________
block_4b_conv_1 (Conv2D)        (None, 384, 20, 20)  1355136     block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_relu_1 (Activation)    (None, 384, 20, 20)  0           block_4b_conv_1[0][0]            
__________________________________________________________________________________________________
block_4b_conv_2 (Conv2D)        (None, 424, 20, 20)  1465768     block_4b_relu_1[0][0]            
__________________________________________________________________________________________________
block_4b_conv_shortcut (Conv2D) (None, 424, 20, 20)  166632      block_4a_relu[0][0]              
__________________________________________________________________________________________________
add_8 (Add)                     (None, 424, 20, 20)  0           block_4b_conv_2[0][0]            
                                                                 block_4b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
block_4b_relu (Activation)      (None, 424, 20, 20)  0           add_8[0][0]                      
__________________________________________________________________________________________________
conv2d_transpose_1 (Conv2DTrans (None, 256, 40, 40)  1736960     block_4b_relu[0][0]              
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 368, 40, 40)  0           conv2d_transpose_1[0][0]         
                                                                 block_2b_relu[0][0]              
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 368, 40, 40)  0           concatenate_1[0][0]              
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 256, 40, 40)  848128      activation_2[0][0]               
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 256, 40, 40)  0           conv2d_1[0][0]                   
__________________________________________________________________________________________________
conv2d_transpose_2 (Conv2DTrans (None, 128, 80, 80)  524416      activation_3[0][0]               
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (None, 192, 80, 80)  0           conv2d_transpose_2[0][0]         
                                                                 block_1b_relu[0][0]              
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 192, 80, 80)  0           concatenate_2[0][0]              
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 128, 80, 80)  221312      activation_4[0][0]               
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 128, 80, 80)  0           conv2d_2[0][0]                   
__________________________________________________________________________________________________
conv2d_transpose_3 (Conv2DTrans (None, 64, 160, 160) 131136      activation_5[0][0]               
__________________________________________________________________________________________________
concatenate_3 (Concatenate)     (None, 128, 160, 160 0           conv2d_transpose_3[0][0]         
                                                                 activation_1[0][0]               
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 128, 160, 160 0           concatenate_3[0][0]              
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 64, 160, 160) 73792       activation_6[0][0]               
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 64, 160, 160) 0           conv2d_3[0][0]                   
__________________________________________________________________________________________________
conv2d_transpose_4 (Conv2DTrans (None, 64, 320, 320) 65600       activation_7[0][0]               
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 64, 320, 320) 0           conv2d_transpose_4[0][0]         
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 64, 320, 320) 36928       activation_8[0][0]               
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 64, 320, 320) 0           conv2d_4[0][0]                   
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 34, 320, 320) 19618       activation_9[0][0]               
__________________________________________________________________________________________________
permute_1 (Permute)             (None, 320, 320, 34) 0           conv2d_5[0][0]                   
__________________________________________________________________________________________________
softmax_1 (Softmax)             (None, 320, 320, 34) 0           permute_1[0][0]                  
==================================================================================================
Total params: 11,222,090
Trainable params: 11,222,090
Non-trainable params: 0
__________________________________________________________________________________________________
2022-05-24 04:17:35,725 [INFO] iva.unet.model.model_io: Loaded weights Successfully for Export
2022-05-24 04:17:35,725 [INFO] root: Using input nodes: ['input_1']
2022-05-24 04:17:35,725 [INFO] root: Using output nodes: ['softmax_1']
2022-05-24 04:17:35,726 [INFO] iva.common.export.keras_exporter: Using input nodes: ['input_1']
2022-05-24 04:17:35,726 [INFO] iva.common.export.keras_exporter: Using output nodes: ['softmax_1']
The ONNX operator number change on the optimization: 144 -> 72
2022-05-24 04:17:40,129 [INFO] keras2onnx: The ONNX operator number change on the optimization: 144 -> 72
2022-05-24 04:17:40,420 [WARNING] onnxmltools: The maximum opset needed by this model is only 11.
2022-05-24 04:17:41,048 [INFO] numba.cuda.cudadrv.driver: init
2022-05-24 04:17:41,191 [INFO] iva.unet.export.unet_exporter: Converted model was saved into /workspace/tao-experiments/unet/kitti_experiment_retrain/weights/model_kitti_retrained.etlt
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 2: Train Id 2
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 3: Train Id 3
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 4: Train Id 4
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 5: Train Id 5
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 6: Train Id 6
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 7: Train Id 7
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 8: Train Id 8
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 9: Train Id 9
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 10: Train Id 10
2022-05-24 04:17:41,203 [INFO] iva.unet.model.utilities: Label Id 11: Train Id 11
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 12: Train Id 12
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 13: Train Id 13
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 14: Train Id 14
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 15: Train Id 15
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 16: Train Id 16
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 17: Train Id 17
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 18: Train Id 18
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 19: Train Id 19
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 20: Train Id 20
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 21: Train Id 21
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 22: Train Id 22
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 23: Train Id 23
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 24: Train Id 24
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 25: Train Id 25
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 26: Train Id 26
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 27: Train Id 27
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 28: Train Id 28
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 29: Train Id 29
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 30: Train Id 30
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 31: Train Id 31
2022-05-24 04:17:41,204 [INFO] iva.unet.model.utilities: Label Id 32: Train Id 32
2022-05-24 04:17:41,205 [INFO] iva.unet.model.utilities: Label Id 33: Train Id 33
2022-05-24 04:17:58,405 [INFO] root: Export complete.
2022-05-24 04:17:58,406 [INFO] root: {
    "param_count": 11.22209,
    "size": 99.92537879943848
}
2022-05-24 13:17:59,775 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.