OOM occured in the process of training efficientdet by AutoML

,

Hello. I got the OOM error message when I trained AutoML efficientdet model.

The dataset I used was from FLIR Thermal Images Dataset and I did data preprocess by this method (Convert to kitti labels)

How should I do to deal with the problem ?

2023-04-18 03:22:35,825 [INFO] root: Saving checkpoints for 0 into /shared/users/80ab3db1-baf9-5608-8a94-f5b86a8cbd59/models/69a4b4f7-069d-4b93-beac-4fd9c7dc6807/0782f6e5-2e4b-4aa4-9e3c-35ed28f6c1c4/experiment_0/model.step-0.tlt.
[GPU 00] Restoring pretrained weights (309 Tensors)
2023-04-18 03:22:38,841 [INFO] root: Pretrained weights loaded with success...

2023-04-18 03:26:24,275 [INFO] root: OOM when allocating tensor of shape [16,64,64,480] and type float
         [[node gradients/zeros_174 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'gradients/zeros_174':
  File "/usr/local/lib/python3.6/dist-packages/iva/efficientdet/scripts/train.py", line 3, in <module>
    __pyarmor_vax_001219__(__name__, __file__, b'\x50\x59\x41\x52\x4d\x4f\x52\x00\x00\x03\x06\x00\x33\x0d\x0d\x0a\x09\x34\xe0\x02\x00\x00\x00\x00\x01\x00\x00\x00\x40\x00\x00\x00\x7a\x10\x00\x00\x00\x00\x00\x18\xf1\x96\xb5\xd4\x0d\x20\xf3\x7e\x8d\xf7\xcd\x3a\x35\x5b\xbd\xc4\x00\x00\x00\x00\x00\x00\x00\x00\x82\x35\x79\x4b\x0d\xd2\x3f\x4e\xa5\x4b\x45\x5a\xb5\x3e\xd1\xb1\xf6\xea\x1f\x14\x4c\xcb\x8e\x04\xcd\xd4\xc6\x8c\x82\x51\xa6\x00\x86\x65\x1b\x12\x68\xc1\x3d\x59\xc3\x9f\x6f\xbe\x07\xc2\x40\x9e\x9c\x6d\x66\x01\x73\x4c\x21\x39\x1f\xa1\x18\x00\x16\x03\x75\x90\xe4\x69\x9e\x2b\x18\xac\xed\xc1\xd6\x69\xc7\x37\x03\x12\x33\x90\x3b\xea\x22\xcb\xe6\xab\x07\xf2\xae\xcf\x27\x0c\xb3\x56\x73\x90\x8b\xa5\xb7\x58\xa9\x86\x1b\x0c\x84\x9d\x32\xf5\xfa\xa4\x9a\xf2\x85\x63\xa0\x47\x7a\xa3\x3a\xa3\x99\x22\x79\x27\x2c\xc0\x4f\x71\x61\x13\x69\x4a\x38\xfc\xdc\x91\x2b\x33\x3a\x85\xed\xb8\x69\x17\xeb\xd9\x33\xfb\x66\x59\x5d\x61\x61\x80\xea\x2c\xa2\x63\xf5\xf5\xe2\x98\xc3\x02\x3a\x04\x30\x29\xc9\xf0\xc4\xfd\xc8\x8b\x39\xab\xdd\xba\x64\x27\x80\x0c\xba\x06\xfe\xb5\x1f\x4f\x52\x14\xd8\x84\xaa\x01\x4e\x84\x6d\x74\xce\x7a\x06\x22\x7e\xad\xc4\x28\xba\x78\xc1\x56\xaf\xb3\x76\x29\xed\x22\x99\x54\x27\x3c\x02\xd8\x81\x07\x25\x05\xb5\xa1\x96\xea\x5c\x4f\x6c\x3b\xba\x76\x7f\xd9\x79\x32\xdf\x2f\x8a\xb4\xd0\x16\x9c\x39\x03\x87\xcf\xa5\xd1\xe2\x1f\x27\xf4\xcb\x2a\x88\x4d\xf1\xf5\xbd\x11\x87\x53\xa4\xe4\x4b\x6b\xd8\x89\xa4\x8b\xb8\xdc\x17\x04\x70\x26\x30\xf2\x73\x29\x34\x02\xc4\xe8\xe6\x34\xbc\xfe\xb8\x3d\x9f\x54\x78\x47\x03\xdd\x04\x18\x77\x4f\xbf\xc8\x93\x80\x8d\x3b\x42\xa0\x29\x5c\x3e\x3f\x5e\xaf\x84\xe7\x73\x0d\x66\x4e\x9f\xea\x81\xde\x37\x79\x12\xec\xb4\xee\x24\xd7\xd1\x25\xae\xb3\xbe\xc3\x96\xca\x4c\x59\xff\xfd\x09\x60\x74\xa2\xc9\x81\x86\x60\x1f\xaf\x83\xbc\x8d\xcb\xc5\x46\xdb\x55\x28\xef\xeb\x4c\xbe\x05\x36\xd5\x4e\xe0\xf2\x37\x72\x2b\x84\x2f\x4d\x8e\xe8\xe3\x3a\xd4\x3c\xc6\xcf\x5f\xd3\xde\x6c\xff\x1e\xda\x5b\xfe\x8c\x35\x9b\xd1\x97\x52\x38\x38\x79\x89\xf9\xcf\x17\xb7\x4b\x9f\x6f\x71\x26\x32\xcd\x0d\x5d\x4d\x3c\xc5\x3c\x9f\xf7\xc4\xae\xc8\x9d\x1e\x35\x1f\x13\xb6\x36\xee\xc7\xe2\x2c\x69\x10\xbf\x20\xa7\xbe\x94\x1c\x31\x8d\x58\x32\xf6\x51\x2e\xa8\x35\xae\xd0\x01\x09\x51\xa0\x76\xe6\x34\x8f\x5f\x3b\x24\xb8\xc1\x19\xd1\xae\x16\xf1\x0c\x20\x6a\x25\xb3\x99\xb2\x1a\x8f\x47\x74\x70\x19\xd2\x2c\x0f\x16\x92\x43\x3e\x8f\xcc\x01\x6a\xc8\xb2\x39\x60\x49\x85\x43\xfc\x77\x39\x96\xa5\x3d\xfa\x40\x43\x02\x3f\xe4\x6d\xc2\xbb\x33\x1f\x44\xff\x1f\x51\x70\x35\x38\xd4\x32\x78\x89\xc6\x49\x10\xd4\x87\x73\xf1\xeb\xc0\x8e\x14\x5e\x02\xa6\x94\x03\x0a\xf7\xee\xfd\x32\x2c\x24\x8d\x12\x71\x10\x93\x81\x73\xd6\xda\x8f\x9a\x08\x93\xcd\x0a\xb6\xf7\xce\x6b\x42\x18\x1f\x15\xc9\x61\x3b\xd5\x96\x64\x79\xc0\x81\x97\xb2\xc6\xc7\x13\xea\x67\x25\x7c\xdf\x8e\x3a\x26\x8a\xd0\x29\xed\x56\x19\xae\xb9\xd5\xbf\xb1\x7e\x64\x9a\xc8\xf8\xdf\xe9\x87\xae\xe4\x81\x24\x2d\x52\x13\x5a\x37\x41\x8a\xea\x42\xf2\x4a\x34\x10\xc6\xb7\x12\x9a\x8f\x87\xa5\xfe\x4e\x8d\xa7\x2c\x6d\xca\x4a\x0d\x5d\x19\x50\x4f\x54\xe3\xc3\x5f\x5f\xe7\x78\x3f\xfa\x81\xb6\x13\x33\x22\xbc\xcf\xf4\xc9\xa4\x7d\xed\x39\xc4\xd7\xf8\x53\xe5\x6c\x9c\xe5\x26\x2e\xfc\x15\xad\x31\x9c\x1d\x80\xe1\xe4\x40\xd4\xb0\x05\xa8\x12\x3c\x84\x78\x48\x25\x6f\x1f\x7e\x40\xe9\xa6\x35\x02\x85\x44\x24\x42\xab\x90\x76\xa3\x8f\xa0\xb8\xfc\x98\xea\x68\xe5\x5c\x89\x03\xf8\xe2\x77\xda\xc6\x94\x5c\x02\x4b\xa7\xd3\x60\x42\x1f\xe9\x06\xcb\xee\x0f\x0a\x6f\x4b\xb8\x34\xf8\x29\x9f\xf3\xb9\xe3\x85\xf6\xb0\x2f\x2e\xe0\x1e\x63\x5b\x31\xea\x47\xe9\xe2\x11\xa6\x6f\xa5\x5e\x20\x41\x0a\x08\x3c\x11\x67\xbb\x16\xe7\xcf\x0d\x9f\x38\x06\x3d\x6e\x11\xaf\x33\x47\x4f\x0d\x00\x1c\x02\x83\xa3\x99\xcf\x44\xfa\xf8\x07\xfc\x19\xc4\x95\x21\x0c\xe6\x97\xea\x55\x6f\x70\x42\x3a\x04\x60\x47\xce\x2d\xe6\xa3\x0c\x0b\x66\xb6\xec\xa8\x74\xf2\x0e\x4c\x3c\x4f\x86\xf4\xd7\xa5\x2c\x0a\x23\x73\x2a\xb5\x72\x24\x57\xe0\x51\x3c\xf0\x47\x50\x6d\xc2\x3c\x8d\x18\x4f\x23\xdc\xb1\xbd\xa5\x2f\xe4\x3f\x8e\xff\xe9\xc2\x36\x66\x61\x30\x60\xbe\x7f\x25\x20\x55\x46\x7f\x6a\x0f\x70\x22\x0d\x73\xb3\xd4\x23\x2b\x7d\x72\x2a\xf6\x5d\xf9\x45\x54\xef\x7d\x6b\xe1\x8a\x26\xd4\x0c\xb8\xb3\x34\x10\xf8\x7f\xab\xfa\xe0\xec\x5c\x51\x55\xbd\x6d\x49\xaa\x81\x40\xf8\xad\x34\x38\xca\xcf\x8b\x98\x17\xeb\xd5\x39\x94\xc4\xca\x2d\x7d\x71\x83\xfb\x39\x26\x28\x87\x95\xbd\x60\x7c\x0b\x34\xd7\xd1\x0a\x7f\x31\x45\xf9\x2d\xba\x30\x7b\x5e\x9a\x1e\x39\x10\xd6\xcb\xbc\xa5\xfc\x19\xab\x39\x78\x67\x01\x41\x07\xb7\x7e\x4a\x1d\xf6\xbd\x42\x55\xda\x88\xc2\xfb\x99\xf6\xfe\x99\x8f\xa3\x47\x1d\x6b\x64\x8d\x01\x9d\xc5\x98\x1c\xa4\xe4\xa3\xcb\x1e\xc8\x62\xf6\xf7\x9e\xc8\x3d\x4e\x1e\xdf\x35\x85\xb2\x35\x9f\xdb\x06\xfa\x2b\xdb\x7b\x38\xdc\x5e\xc6\xd5\x4a\xa6\xd9\x3a\x03\x7f\x38\x4e\xf5\x87\x8c\x05\x73\x23\xba\x92\x9f\xfa\x8f\x12\xa5\x74\xc6\x6d\x28\xf7\xd6\x8f\xad\x3a\x9f\x0c\x09\x9a\x4f\x8f\x99\x59\xfb\x8b\xef\x39\xe1\x29\xe2\x1e\x0a\x9b\x66\xea\x86\xf2\xae\x19\x9f\x30\x85\x2b\x7b\x3a\xcb\x22\xee\xc0\x6b\x58\x0e\x98\x11\x0a\xfd\x0d\x05\xc1\x8e\x43\xba\x76\x4e\xf0\x95\xff\x60\xc1\xa5\x15\x2a\x33\x72\x56\x9b\xfa\x9f\xbb\x05\x3e\x77\xf8\xe7\xe4\xa6\xd5\x0f\xc9\xd2\xea\xf5\x40\x83\x54\x22\x0b\x04\x2c\xf9\xfa\x4f\xeb\xc0\x5a\xf6\xc6\x90\xce\x3f\x48\xf7\xb8\x69\x0a\xb4\x97\x46\x28\xf3\x92\x76\x15\x51\xd7\x32\x21\x34\xff\x5f\x25\xb6\xde\x0d\xb9\xd2\x42\x13\xc7\x56\x1e\x17\xfa\xc6\x89\x37\x17\x10\xff\x87\xb1\xee\x4b\x43\xa0\x38\xc0\xca\x69\x5b\xf2\x72\x0e\x69\x01\x09\xad\xd4\x5e\xf1\x65\xd5\xe6\x61\x71\x52\xd9\x2b\x2f\xbe\x21\xfb\x14\x0e\x40\x38\xc2\xa7\xe6\x63\x75\x7a\xcd\xe0\x4b\x1c\x43\x2e\x1a\x8e\xea\x53\x1a\x26\x96\x95\x9c\xbe\x25\x1e\x9a\xe6\xf4\x22\x08\xb9\xc2\xa4\x80\xab\xd1\x54\xa0\x46\xc3\x41\x48\xb8\x96\x68\xa7\x08\xc7\x6e\xe6\xc1\x48\x29\xf4\xe9\x70\x58\x31\x79\xbd\x19\x68\x9b\x1d\x40\xa9\xc0\x8b\xcb\x83\xbc\x93\x10\x4f\x9f\xfb\xec\x82\x14\xb1\xc9\x19\x2e\xb3\x3b\x4a\x6d\xee\x02\xb6\x48\x43\xca\x14\xec\xcf\x46\xf3\x03\x75\xa2\xa2\x1b\x1e\x6e\x02\xbc\xd8\x19\x6f\xb5\x1f\x9a\xe7\x00\x33\xe0\x68\x56\xa5\xf7\x9d\x81\xf5\xa1\x0a\xa8\x80\x31\x51\xcb\x5f\x31\xb1\xf2\x5b\x91\x9b\x98\xac\x66\x10\x06\xc5\x27\xd9\x1b\xef\xa9\x5f\xba\x98\x7c\xf3\x77\xd5\x7b\x27\x98\xf3\xf6\x68\x6a\x92\x99\x22\xf1\xb6\x45\x6f\x83\x44\xee\x30\x0a\x39\x72\xcb\x28\x51\x9d\x93\x09\x10\xf8\xec\xa8\x36\xb1\x1a\x2b\x1a\x8c\x9e\x8d\x2d\x5b\xa0\x1b\x97\x27\x65\x00\x30\x14\xb3\x87\x75\x1d\x72\xb2\x0c\xdc\xa6\xc8\x04\xac\xd8\xa0\xb7\xc5\x46\x56\xa2\x82\xc4\x92\x1f\x11\x88\xa5\x87\x34\x4c\x41\xf1\xd0\x0b\xc5\xcd\xea\xf6\x8f\xc6\x3b\x65\xd1\xa6\x4b\x9b\xd3\xe8\x6a\x96\x87\xb5\xac\x64\xcd\x6d\x6f\x4d\xeb\x73\x6e\xf7\x05\x6c\x59\xfc\x8d\xba\x05\x6f\x75\xca\x23\xf0\x76\xed\x4f\x30\x7c\xe3\xbf\x0c\xad\xee\x89\x42\x0e\xec\xaa\x3f\x8a\xe7\xc9\x94\xa9\x68\x39\x2a\x15\xf2\x21\xcc\xf0\xed\x5c\xa9\xed\x72\x8f\xf7\x6a\x12\x66\x86\x0d\x89\x56\xdc\x24\x78\xe6\x5a\x99\xed\x40\xff\x9c\x9d\x28\xde\xe7\xd3\x59\x4e\xe5\x57\x1b\x23\xaf\xe9\xe1\xd4\x58\xfd\xc9\x1f\xaa\xaf\x23\xa3\xc4\x54\xb0\xbb\x48\xa8\x80\xd9\xdd\x1d\x1d\xd9\x89\x28\xa5\xa8\x26\x4c\xfc\x94\xa4\x1b\xde\x19\xe6\xb0\xd1\x4b\xf5\x47\xed\x28\xd3\x81\x63\xe8\x9f\x52\x26\x32\x75\xe0\x46\xb2\x14\x02\x48\x6d\xfd\x4e\x3f\xb2\x91\x6c\xa9\xa8\xbe\xf5\xc8\x6d\x96\x31\xf6\xba\x80\xf2\x15\x6b\xee\x41\xfe\x19\xfb\x39\xd2\x4d\x4b\x3f\x67\x20\xbc\xe3\x4d\x57\xc4\x4c\xb6\x7c\x57\x5b\x12\x1e\x75\x7c\xb3\x67\x59\x66\xea\x59\x11\x96\x31\x06\xd7\xdd\xc4\x2f\x29\x1f\x1c\x3a\xa7\x37\xe9\x7b\x21\xe4\x27\xe9\x32\x44\xa9\xc6\xd7\xb6\x39\xae\x61\xa3\x5a\x65\x25\xae\xb9\x0d\x80\x3f\xe7\xdf\xc1\xcc\x6b\x28\x55\xb8\xe6\xcf\x92\x2a\x5b\x3d\x22\x43\xd8\x84\x15\x07\xee\xa3\x4a\x27\x78\x5a\x26\x86\x39\xa9\x30\x52\x26\xe4\xbc\xb8\xd6\xc3\xfa\xc4\xb6\xa7\xa7\xa2\x92\x04\x2f\x55\x6b\x4c\x28\x28\x28\xb6\x00\xe9\x20\x3d\x4a\xb7\xa3\xa4\x00\x45\x9b\xcb\xa2\x1b\xad\x75\xbc\x67\xd1\x0a\xcf\x9a\x43\x43\x15\xde\x05\xdf\x03\x11\xcd\x22\x8e\xa7\x03\xeb\xf1\x72\xc4\xad\xf0\x9a\x34\x29\x28\xd1\x79\x3c\x98\xc2\xeb\xe2\x45\x35\xb9\x22\xfe\x18\xff\x5b\x56\x4a\xe6\x41\x3a\x71\x12\x9b\x45\x2c\x9e\x08\x58\xcb\xe4\x22\xaf\x4c\xb5\x43\xae\x08\x7a\x55\x58\xf8\xd8\x50\x20\x13\x1e\x19\x8e\x11\x81\xba\x55\x08\x6c\x58\xff\xc1\x28\x8f\xd8\xf8\x4d\x36\xac\x55\xfa\x11\x25\x9f\x71\x77\x59\x6b\x58\x70\x52\xb3\x56\xc0\xfd\x57\x32\xc0\xc1\xe8\x34\xd7\x11\x58\x9e\x4c\x84\x81\x67\x2e\x02\xfa\xc7\xdf\x52\x8c\x9f\x17\x2d\xfb\x80\x0b\xcb\x45\x17\xca\xcc\xe2\x94\x95\x0f\x71\x79\xf0\x1c\xe7\xb5\x01\x7f\xe4\x83\x29\xcc\xc8\xac\x08\x51\x3c\x66\x29\x97\x0e\x1a\x68\xb2\x89\x13\xaa\xb3\xfc\x8f\x7e\xc9\x64\xae\x34\xe8\x2d\x42\xb3\x87\x72\xdc\x6e\xd0\xf4\xea\x6e\x87\x88\x76\xc4\x57\xa1\xae\xaa\x48\x56\x3c\x0e\x77\x0a\x8b\x5f\xe6\x18\x94\x52\x89\x74\xb5\xee\xcf\x9f\x3e\x51\x1e\x61\xb4\xce\xef\xaf\xcb\xb0\x3c\x1d\x7b\x70\x5f\x68\xcf\x4b\x36\xa3\xa8\x76\xb4\x93\x44\xdd\x12\x5a\x97\xee\xfb\x32\x6a\x03\xa9\x32\x55\xb4\x27\x60\x47\x48\x50\x40\x57\x2a\x2c\x1a\x64\x79\x5f\x81\x2e\x5a\x8d\xf4\x20\x91\xbe\x54\xfb\xc3\x7f\x86\xd1\x91\x10\x44\x38\x9e\xca\x81\x7f\xdf\x58\xae\xda\xbc\x63\x2c\xbe\x95\x4b\x24\xdd\x2e\xac\xa0\x91\xa9\x3b\x42\x44\xd1\x9c\x74\xce\xf9\x98\x22\x5b\x12\x41\x27\x82\xc4\x56\xeb\x2c\x85\x95\x70\x65\x3f\x63\x7e\xb8\xa5\x0f\xe5\xbf\xaa\xc6\x01\xdf\xbc\x7d\x71\x00\x22\x05\x10\x49\xa8\x8a\x51\xe9\x22\x47\x2e\xd7\x83\xdb\x01\x95\x38\xe5\xe7\x8b\x76\x77\x47\x09\x31\x33\xbe\x47\x36\xe8\x5b\x0f\x47\xe8\x58\xc9\x23\xe6\xd4\x57\xfb\xc6\x09\x5b\x9b\xdd\xec\x14\x47\x2a\x39\xeb\xdf\x56\xea\xd6\xc1\xeb\xda\x68\x04\xd8\x01\xdf\x13\x52\x21\xb9\xbe\x26\xb8\x0a\xaf\x58\x3f\x6f\xd2\x9a\x40\x6b\x1b\xfd\x7e\x01\xbb\x04\x14\x6e\x8a\x89\x7f\x97\x5b\xc4\xb6\xcf\x82\x6f\x00\x8c\x96\x20\x26\xef\x6e\xa5\xd5\x26\x2c\x8b\x5e\xd0\xb6\x59\x2a\xca\xbb\xa6\xeb\xea\x81\xad\x65\xc0\xb7\xed\xed\x99\xb3\x9c\x29\x15\x91\xcd\xab\x2e\xe5\x67\x0a\x76\x3c\x97\x32\x3e\x42\x94\x47\x9a\xc1\x21\x4b\xa2\x09\xde\xd4\xd2\xaf\xab\x4e\xd8\x78\xa6\xf2\x31\xe4\xe5\xea\x7f\x56\x36\x11\x75\x17\x6f\x36\x17\xfa\x49\x18\x89\x0e\x04\x93\xc3\x98\x77\x71\x1b\x7f\xcc\x00\x78\x90\x81\x6e\xf5\x5e\x50\x5a\xa7\x36\xbc\x39\xfd\x28\xdd\xfd\xea\x64\xee\x7b\x6b\xc8\x4b\x96\x53\x57\x0e\xf9\x3c\xb6\x39\xe7\xea\xce\x94\xfe\x49\x28\xfb\x94\x33\x43\x92\xd7\x3b\x40\x98\x58\x23\x90\xdc\x8d\x9d\x26\x84\xf8\x3d\xb5\xb9\x97\xdb\x46\x1e\x07\x09\xee\xe9\x18\x0d\x37\xdb\x7f\x50\x5a\x64\xf6\x72\x94\xb8\xfc\x4d\x88\xe8\xb6\x05\xd3\x28\xaf\x55\x8a\x1e\x16\x10\x65\xbd\x39\x5c\x43\xd1\x6b\x9c\xd6\x70\x5e\xc4\xd9\xe2\x6f\x99\xe3\x70\x24\x7e\x97\xad\x04\x9b\xc4\x24\x14\xb5\x98\xbc\x21\x03\x76\x1f\x10\xe6\xe0\x6b\x28\x58\x67\x83\xdf\xdd\x80\x6e\xb7\x65\xf3\x8a\xee\x95\x25\x1b\x6f\xa1\xbb\x60\x3f\xdd\x01\xb7\xb7\x76\xe2\xb2\x3a\x5b\xcf\xd5\xe3\x90\x7f\x2e\x72\xdd\x3a\x2f\x0c\x35\x6c\xae\xeb\x80\xd5\xa1\x6c\x20\x51\x05\xc4\xb4\x0d\xa5\x17\xc5\xa6\x21\x6d\x4b\x6e\xbc\xb6\x02\x44\xce\x75\xb4\x9e\xd6\xc8\xeb\xf1\x5e\xc0\xe4\x8e\xae\x79\x5f\xc0\xe5\x0d\x4d\xff\x69\xf3\xc5\x64\x4a\x9e\xf6\x87\x02\x45\xe7\xb6\xe1\x00\xb4\x33\x25\xfa\x4c\x5a\x87\xb1\x8f\xc5\x97\xc4\xb6\xce\xf9\x35\x5d\x84\xe2\xbb\x12\x23\xce\x04\x9f\xee\x87\x38\x67\x4b\x5b\x52\x6b\x47\x63\x4d\x09\xa3\x13\x50\xf2\xf1\x7d\x2d\x1b\x93\x32\x18\x83\xd9\x8e\xf9\x20\x0f\x8c\x3e\x35\xe4\x10\xc2\x75\x5a\xd6\xfa\x2b\x5b\x14\x4b\x99\xa7\xfc\x53\x20\xac\xeb\x49\x74\x82\xde\xe0\x35\xc1\xe4\xd2\xc7\x4c\x42\xab\xdb\x38\x84\x86\x45\x36\x77\xa1\x3f\x73\x98\xb6\x35\xc9\xd4\x33\xf5\xd7\xf8\x18\x38\xe2\xf8\xff\x5b\x34\xd9\xac\x93\xb5\x1c\xbf\xdd\xe9\x92\x53\x26\xc4\xd2\xd2\x0e\xbd\x37\xe8\xa8\x39\x80\xb8\x61\x70\xd6\xe7\x44\xab\x9d\x91\x53\xcf\x1d\x64\xc5\xf5\xe4\x16\xda\x54\x82\x29\x51\x02\x20\xa9\x20\xd6\x07\xcc\x23\x97\x82\xd4\xf4\x65\x5b\x9d\x25\x24\x8b\xae\xb4\x74\x85\x6e\x8e\x89\x2b\xb4\x1b\x85\xea\x8a\x01\xa9\x24\x15\x31\x57\xf0\xae\x3d\x7c\xca\x83\xc9\x7a\xd7\x69\x93\xae\x15\x86\xc5\x93\xa8\xda\xf3\x0d\x80\x52\x24\x5d\xe1\xf1\x17\xfd\xdc\x64\x14\x81\x5f\xf5\x82\xb3\x69\x7a\xc3\xd6\x24\xef\xec\x2d\x28\x53\xb9\xf8\xc8\xdb\xc8\xbe\xe0\xe5\x51\x72\x47\x9e\x78\xe8\xcc\x35\xb1\x61\x32\xb8\x51\x4b\x41\x31\x26\x68\xfb\xda\x09\x8d\x1d\xe9\xcc\x94\xed\x3a\xd7\x3f\x76\x07\xa3\x3b\x89\x16\xef\x9d\xd9\x07\xe0\x66\xc9\x8f\xbf\x19\x30\xe1\xfc\xfe\x73\x97\xea\x0d\xbb\xeb\x29\xba\xb1\xa7\x93\x0a\x10\xff\xc9\x05\x7a\xbb\xc4\x74\x2a\x83\x26\xc4\xc9\x41\x82\xc0\xbd\x41\x91\x48\xe7\xee\x35\x43\x5f\x2f\x8b\xe3\x6c\x3f\x99\x9d\xd5\x9f\x33\x9a\x21\x01\xd5\x4c\x2e\xef\xd8\x52\x91\x7a\x06\x8f\xe3\x2b\xfc\xeb\xa0\x42\xd0\xfd\x02\xd0\xa9\xd5\x84\xf3\x22\x20\xb1\xb6\x83\x9f\x7d\x10\x10\x55\x62\x5d\xcf\x01\x54\x65\x95\xe5\x50\xb5\x03\xe1\x1c\x2b\x7f\x3a\xc7\x48\x01\x13\xc5\x73\x2c\x8e\xe5\xaf\x4f\x1c\x4e\x21\x0b\x22\x6e\xcb\xb8\xcf\x54\x6a\x8d\x56\xba\x11\xa3\x2b\x76\x4e\xdb\xef\x27\x4e\x15\x1f\xc6\xa1\x2c\x33\xc3\x36\x5c\xaa\x3e\x2d\xf0\x71\x9e\x1c\xa2\xf3\x07\x08\xcb\x01\x9b\xcd\x22\x1f\x17\x07\x28\xf6\x0a\x63\xd0\x5b\xba\xea\x12\x80\x0a\x9a\x1b\x01\x48\x43\x6a\x01\xcb\xbe\xca\x34\x15\x7b\x13\xbe\xee\x2a\xd2\x85\x2a\x03\x51\x3b\x9b\x22\x32\x1a\xa5\x30\xec\xdf\x70\x9d\x77\x4c\x5c\x90\x41\x43\x3c\x01\xe6\x77\x48\x2f\x46\x41\x02\x3d\x44\x01\xe7\x42\x08\xb1\xf3\x95\x96\x90\x67\xcb\x4d\xda\x7f\x70\x82\xe9\x66\x4a\xe0\x4a\xe7\x21\x07\x10\xec\xf6\xa2\x79\xb1\xd6\xa3\x2a\x7d\xe9\x09\x8a\x30\x15\x1f\xdb\x70\x4a\xa3\xac\xcf\xaf\xa7\x19\xde\x98\x68\xa5\x9f\x24\xa7\x3a\xf5\xf0\xc7\x8d\x31\x05\xc3\x23\x8a\x7e\x7d\xce\x47\x76\x87\xaf\xb0\xc2\x97\x5e\x78\xdc\x90\xf7\xc2\x4d\x8d\x77\x87\x29\x14\x81\x06\xd0\x7b\x17\xa5\xfd\xe0\x8e\x04\xee\x81\x69\x72\xd1\x3e\xe9\xe5\x76\x83\xa2\x19\x42\x55\xb0\xf5\x3e\xdb\xc0\xda\x41\x67\xc1\x4a\x29\xae\x69\xcd\xf8\xea\x1b\xfc\x25\xd7\x98\x0f\x3b\xb1\x23\xe7\xce\x29\xc4\x86\xd0\x67\x0d\x8d\x84\xf5\x68\xc4\xa6\xfc\x77\x1e\xdf\x10\xb5\x96\x84\x0e\xc2\xb0\x8a\xcb\xf5\xc3\x6b\x13\x1b\x59\x9a\x46\x43\x88\xc1\xe6\x3a\x02\x50\xf5\xbc\x6c\x88\x9f\x81\x4f\x7a\xa4\xfe\x34\xdd\x18\xf2\x69\x85\xe2\xd1\xbd\x0b\x3a\xc3\xfd\xc5\x8c\xdc\xdf\x72\x49\xa5\x7b\x9e\x5e\x49\xb4\xf1\x34\x04\x92\xac\x1b\x18\xf1\xfc\x6f\x2c\xa2\xf2\x0c\x7b\x25\xfe\x02\x77\x03\xef\x8e\x38\xf2\xdc\x29\xd0\x78\x6e\xc5\xbb\x16\xc2\xc8\xca\xd9\x68\xfb\x41\x3a\x4d\x77\x91\x58\x93\x96\xdd\xce\x6b\x2d\x12\xcf\x52\xd8\x59\x89\x05\x32\x7b\xeb\xdb\xfd\x86\x5a\xde\xb4\x40\xd6\xe5\x23\xc9\xd3\x2c\x46\xbc\xef\xe4\xb8\x54\x4c\xad\x6a\xf3\xda\x31\xe3\x29\xe1\xe3\xa5\xa0\x1f\x90\xdc\xe8\x62\xfe\xcb\xb1\x0a\xaa\x9e\x48\xb1\x6c\x9a\x97\xad\xc0\xf3\x6e\xec\xc6\x07\xfc\x00\x54\x70\xdb\xbd\xe5\x24\x33\xa3\x18\xd1\xf9\x0e\x2a\x74\xad\x1b\xeb\x91\x66\x55\xe7\xf5\xf2\xef\x45\x45\x9e\x7a\xcb\x43\xa3\x10\x42\xa1\x1f\x6d\x17\x16\x98\x53\xc3\xb0\x70\x29\x1e\xdb\x99\x13\xfe\x77\x12\x9d\x2c\x94\xfa\x56\x58\xf5\x27\x5f\x52\x40\xeb\x2b\x36\x89\xc9\xc6\x12\x88\x04\x0b\x64\xa1\x35\x53\xf6\x1c\x95\xcc\x36\x97\xce\xd6\x6d\x62\x1c\x51\x1d\xc9\x8f\xc4\xc3\x9d\xaf\xfa\xcd\x0e\x3b\xeb\xc0\xf6\xd5\x24\x84\xd2\xde\x51\x58\xfd\x46\x97\x70\xc3\x94\x5a\x6b\x18\x01\xb1\x8a\x19\x54\x2a\xd8\xd9\x0b\xab\xd2\x37\xfe\xca\x77\xa6\xc7\x08\x31\x76\x5b\xe6\x14\x0f\x16\xbc\xe7\xf6\x1b\x96\xe4\x55\x62\x56\xd0\x9d\x1e\xa6\xd8\x10\x26\x22\x02\x1d\x07\xa1\xd7\xaa\x87\x86\xbd\x7c\x45\x29\xc8\x7c\x91\x35\x6d\x2d\xc9\x3c\x9b\x59\x2d\x0c\xec\xf4\xc0\x0d\xf5\xb2\xdf\x06\xb2\x23\x4e\xb0\x5e\x00\x67\xfa\xed\xd6\x19\x73\xa6\xe5\x61\x73\x26\xce\x24\x13\xbb\xbf\xfe\xc3\x72\x8d\x26\xae\x12\x26\x80\xd1\x27\x2a\x04\x34\x60\x67\xe8\x3c\x2b\xa2\xbd\xb4\x20\xb0\x10\x4f\x89\xf7\xd3\xdc\x9e\x6d\x49\x88\xbb\x5f\x2a\xbf\x79\x7f\x52\x35\x23\x48\x8e\x0f\xb7\xb2\xc1\xc2\xe6\x43\x2e\x7b\x35\x83\x9d\x5f\xde\x2c\x6d\xff\xa6\xb1\x01\x8b\x38\x1c\x56\xbb\x07\xe3\xb8\x90\xde\x08\x80\x30\x50\x3f\x66\x0c\x37\xb8\x34\x52\xa2\x37\x68\x3f\xce\xbe\x59\x3e\x87\x37\xba\x40\xfb\xaa\x28\x28\x54\x66\x7b\xc0\x32\x04\x9d\x7d\x81\x7c\xc7\xd2\x13\x07\x38\xf4\x7b\x58\x86\x4f\x22\x4c\x9b\xfe\xc2\x45\xf8\xf0\x1d\xb7\x21\xcc\x09\x55\x34\x27\x5c\xae\xf5\x15\xe6\x3b\xcf\x13\x0f\x5f\xce\xfb\xb2\x96\xe1\x63\xa3\x74\x74\x78\x42\x39\x95\x16\x7e\x31\xa9\x01\x95\x63\x20\x10\xd7\xcd\xa3\x92\x83\x9f\x54\x16\x4a\x06\x79\x80\x73\xd2\x55\x90\x7f\x58\x83\xcf\x77\xcf\xec\x9d\x5c\x2f\x5f\x8e\xe7\x02\x5a\xd7\x94\x05\xe8\x78\x76\x73\x94\xe5\xed\xd9\xbf\x4f\x82\xbb\x08\x0d\xa4\x6f\x71\xb6\xc5\x5b\xee\x65\x26\x4b\xf7\xc2\xdf\x08\x44\xa1\x1d\x8a\x7b\x1c\xb0\x82\xa7\x6a\xdd\x35\xad\xb3\x6f\x1a\xf2\xd2\x80\x59\x3b\x6f\xe6\xcc\x7f\x53\xed\x19\xc3\x68\xaa\x19\xe5\xa1\xcb\xca\xca\x69\x2a\x13\x08\x85\xdc\xcd\xe2\x90\x66\xae\xf7\x2b\xd6\xbb\xef\x9f\x14\x02\x5a\xbd\x92\x38\x37\x87\x65\xde\x82\xf8\xac\xf4\x03\xf9\x23\x51\x3a\xdd\xec\xc8\xe9\x2c\xb6\x84\x37\x4b\xff\xf9\xfd\xb1\x49\xdf\x3b\x95\xcf\x1f\x5a\x5b\x6b\x6a\x0d\xa3\xd2\xe4\x36\x41\x76\x07\x4c\xed\x72\x0f\xd7\x32\x0b\x1e\x04\xff\xdf\x1f\xb0\x4e\x25\x2c\x76\xda\x3c\xe5\xfd\xfc\x29\x13\xc1\x6c\x8f\x4c\x0e\x8a\x01\x74\xab\x1f\xf8\x2e\x74\xdb\x57\xaf\x87\x7a\x72\xe7\xf0\xb4\xc9\x14\xf3\x62\x15\xba\xaa\x95\xcd\x1f\x09\xbf\xc2\x39\x11\x21\xdc\xe4\xee\x3b\x91\x7c\xf6\x3f\xa4\x22\x43\x39\x90\x3d\xa4\xa4\xb8\xd5\x97\x4b\xdd\xac\x9b\xba\x16\xe0\xba\xef\xa1\x21\xd0\xb8\x6b\x89\x57\x54\x94\xf2\x32\xa8\xde\xf3\x58\xe0\x6d\xee\xb7\x85\x92\xfe\x66\x1a\x7d\xad\x09\x49\x6e\xbc\x89\x2b\x3c\x5a\xaa\x76\x30\xf2\x0a\x0c\x92\xe8\x7d\x7b\x49\xc9\xba\xf2\x9f\xde\x9a\x45\x8c\xe2\x31\xf5\x55\x3d\x9e\xb2\x3e\xf4\xdc\x46\x97\x15\xcb\x9b\x2b\xbf\xba\x26\x15\x54\x2b\x8d\x32\x9d\x4b\x14\x02\xf7\x3b\xea\x60\x46\xf9\xe3\x64\x59\x08\x64\xe7\xe8\x5b\x34\x8a\x12\xcb\xa0\x9f\x3e\x26\x7a\x63\x51\x07\x6b\x1c\x47\xb0\x4a\x1b\xaa\xe0\xdd\x3f\x45\xe2\x09\x47\x51\xf7\xf6', 2)
  File "<frozen iva.efficientdet.scripts.train>", line 141, in <module>
  File "<frozen iva.efficientdet.scripts.train>", line 71, in main
  File "<frozen iva.efficientdet.scripts.train>", line 91, in run_executer
  File "<frozen iva.efficientdet.executer.distributed_executer>", line 349, in train_and_eval
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1191, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "<frozen iva.efficientdet.models.det_model_fn>", line 657, in efficientdet_model_fn
  File "<frozen iva.efficientdet.models.det_model_fn>", line 547, in _model_fn
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/experimental/loss_scale_optimizer.py", line 123, in compute_gradients
    grad_loss=grad_loss)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 496, in compute_gradients
    gradients = self._optimizer.compute_gradients(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/optimizer.py", line 537, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
    unconnected_gradients)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 694, in _GradientsHelper
    out_grads[i] = control_flow_state.ZerosLikeOutsideLoop(op, i)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_state.py", line 805, in ZerosLikeOutsideLoop
    return array_ops.zeros(zeros_shape, dtype=val.dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 2350, in zeros
    output = fill(shape, constant(zero, dtype=dtype), name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 171, in fill
    result = gen_array_ops.fill(dims, value, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 3602, in fill
    "Fill", dims=dims, value=value, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [16,64,64,480] and type float
         [[{{node gradients/zeros_174}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "</usr/local/lib/python3.6/dist-packages/iva/efficientdet/scripts/train.py>", line 3, in <module>
  File "<frozen iva.efficientdet.scripts.train>", line 141, in <module>
  File "<frozen iva.efficientdet.scripts.train>", line 84, in main
  File "<frozen iva.efficientdet.scripts.train>", line 71, in main
  File "<frozen iva.efficientdet.scripts.train>", line 91, in run_executer
  File "<frozen iva.efficientdet.executer.distributed_executer>", line 349, in train_and_eval
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [16,64,64,480] and type float
         [[node gradients/zeros_174 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'gradients/zeros_174':
  File "/usr/local/lib/python3.6/dist-packages/iva/efficientdet/scripts/train.py", line 3, in <module>
    __pyarmor_vax_001219__(__name__, __file__, b'\x50\x59\x41\x52\x4d\x4f\x52\x00\x00\x03\x06\x00\x33\x0d\x0d\x0a\x09\x34\xe0\x02\x00\x00\x00\x00\x01\x00\x00\x00\x40\x00\x00\x00\x7a\x10\x00\x00\x00\x00\x00\x18\xf1\x96\xb5\xd4\x0d\x20\xf3\x7e\x8d\xf7\xcd\x3a\x35\x5b\xbd\xc4\x00\x00\x00\x00\x00\x00\x00\x00\x82\x35\x79\x4b\x0d\xd2\x3f\x4e\xa5\x4b\x45\x5a\xb5\x3e\xd1\xb1\xf6\xea\x1f\x14\x4c\xcb\x8e\x04\xcd\xd4\xc6\x8c\x82\x51\xa6\x00\x86\x65\x1b\x12\x68\xc1\x3d\x59\xc3\x9f\x6f\xbe\x07\xc2\x40\x9e\x9c\x6d\x66\x01\x73\x4c\x21\x39\x1f\xa1\x18\x00\x16\x03\x75\x90\xe4\x69\x9e\x2b\x18\xac\xed\xc1\xd6\x69\xc7\x37\x03\x12\x33\x90\x3b\xea\x22\xcb\xe6\xab\x07\xf2\xae\xcf\x27\x0c\xb3\x56\x73\x90\x8b\xa5\xb7\x58\xa9\x86\x1b\x0c\x84\x9d\x32\xf5\xfa\xa4\x9a\xf2\x85\x63\xa0\x47\x7a\xa3\x3a\xa3\x99\x22\x79\x27\x2c\xc0\x4f\x71\x61\x13\x69\x4a\x38\xfc\xdc\x91\x2b\x33\x3a\x85\xed\xb8\x69\x17\xeb\xd9\x33\xfb\x66\x59\x5d\x61\x61\x80\xea\x2c\xa2\x63\xf5\xf5\xe2\x98\xc3\x02\x3a\x04\x30\x29\xc9\xf0\xc4\xfd\xc8\x8b\x39\xab\xdd\xba\x64\x27\x80\x0c\xba\x06\xfe\xb5\x1f\x4f\x52\x14\xd8\x84\xaa\x01\x4e\x84\x6d\x74\xce\x7a\x06\x22\x7e\xad\xc4\x28\xba\x78\xc1\x56\xaf\xb3\x76\x29\xed\x22\x99\x54\x27\x3c\x02\xd8\x81\x07\x25\x05\xb5\xa1\x96\xea\x5c\x4f\x6c\x3b\xba\x76\x7f\xd9\x79\x32\xdf\x2f\x8a\xb4\xd0\x16\x9c\x39\x03\x87\xcf\xa5\xd1\xe2\x1f\x27\xf4\xcb\x2a\x88\x4d\xf1\xf5\xbd\x11\x87\x53\xa4\xe4\x4b\x6b\xd8\x89\xa4\x8b\xb8\xdc\x17\x04\x70\x26\x30\xf2\x73\x29\x34\x02\xc4\xe8\xe6\x34\xbc\xfe\xb8\x3d\x9f\x54\x78\x47\x03\xdd\x04\x18\x77\x4f\xbf\xc8\x93\x80\x8d\x3b\x42\xa0\x29\x5c\x3e\x3f\x5e\xaf\x84\xe7\x73\x0d\x66\x4e\x9f\xea\x81\xde\x37\x79\x12\xec\xb4\xee\x24\xd7\xd1\x25\xae\xb3\xbe\xc3\x96\xca\x4c\x59\xff\xfd\x09\x60\x74\xa2\xc9\x81\x86\x60\x1f\xaf\x83\xbc\x8d\xcb\xc5\x46\xdb\x55\x28\xef\xeb\x4c\xbe\x05\x36\xd5\x4e\xe0\xf2\x37\x72\x2b\x84\x2f\x4d\x8e\xe8\xe3\x3a\xd4\x3c\xc6\xcf\x5f\xd3\xde\x6c\xff\x1e\xda\x5b\xfe\x8c\x35\x9b\xd1\x97\x52\x38\x38\x79\x89\xf9\xcf\x17\xb7\x4b\x9f\x6f\x71\x26\x32\xcd\x0d\x5d\x4d\x3c\xc5\x3c\x9f\xf7\xc4\xae\xc8\x9d\x1e\x35\x1f\x13\xb6\x36\xee\xc7\xe2\x2c\x69\x10\xbf\x20\xa7\xbe\x94\x1c\x31\x8d\x58\x32\xf6\x51\x2e\xa8\x35\xae\xd0\x01\x09\x51\xa0\x76\xe6\x34\x8f\x5f\x3b\x24\xb8\xc1\x19\xd1\xae\x16\xf1\x0c\x20\x6a\x25\xb3\x99\xb2\x1a\x8f\x47\x74\x70\x19\xd2\x2c\x0f\x16\x92\x43\x3e\x8f\xcc\x01\x6a\xc8\xb2\x39\x60\x49\x85\x43\xfc\x77\x39\x96\xa5\x3d\xfa\x40\x43\x02\x3f\xe4\x6d\xc2\xbb\x33\x1f\x44\xff\x1f\x51\x70\x35\x38\xd4\x32\x78\x89\xc6\x49\x10\xd4\x87\x73\xf1\xeb\xc0\x8e\x14\x5e\x02\xa6\x94\x03\x0a\xf7\xee\xfd\x32\x2c\x24\x8d\x12\x71\x10\x93\x81\x73\xd6\xda\x8f\x9a\x08\x93\xcd\x0a\xb6\xf7\xce\x6b\x42\x18\x1f\x15\xc9\x61\x3b\xd5\x96\x64\x79\xc0\x81\x97\xb2\xc6\xc7\x13\xea\x67\x25\x7c\xdf\x8e\x3a\x26\x8a\xd0\x29\xed\x56\x19\xae\xb9\xd5\xbf\xb1\x7e\x64\x9a\xc8\xf8\xdf\xe9\x87\xae\xe4\x81\x24\x2d\x52\x13\x5a\x37\x41\x8a\xea\x42\xf2\x4a\x34\x10\xc6\xb7\x12\x9a\x8f\x87\xa5\xfe\x4e\x8d\xa7\x2c\x6d\xca\x4a\x0d\x5d\x19\x50\x4f\x54\xe3\xc3\x5f\x5f\xe7\x78\x3f\xfa\x81\xb6\x13\x33\x22\xbc\xcf\xf4\xc9\xa4\x7d\xed\x39\xc4\xd7\xf8\x53\xe5\x6c\x9c\xe5\x26\x2e\xfc\x15\xad\x31\x9c\x1d\x80\xe1\xe4\x40\xd4\xb0\x05\xa8\x12\x3c\x84\x78\x48\x25\x6f\x1f\x7e\x40\xe9\xa6\x35\x02\x85\x44\x24\x42\xab\x90\x76\xa3\x8f\xa0\xb8\xfc\x98\xea\x68\xe5\x5c\x89\x03\xf8\xe2\x77\xda\xc6\x94\x5c\x02\x4b\xa7\xd3\x60\x42\x1f\xe9\x06\xcb\xee\x0f\x0a\x6f\x4b\xb8\x34\xf8\x29\x9f\xf3\xb9\xe3\x85\xf6\xb0\x2f\x2e\xe0\x1e\x63\x5b\x31\xea\x47\xe9\xe2\x11\xa6\x6f\xa5\x5e\x20\x41\x0a\x08\x3c\x11\x67\xbb\x16\xe7\xcf\x0d\x9f\x38\x06\x3d\x6e\x11\xaf\x33\x47\x4f\x0d\x00\x1c\x02\x83\xa3\x99\xcf\x44\xfa\xf8\x07\xfc\x19\xc4\x95\x21\x0c\xe6\x97\xea\x55\x6f\x70\x42\x3a\x04\x60\x47\xce\x2d\xe6\xa3\x0c\x0b\x66\xb6\xec\xa8\x74\xf2\x0e\x4c\x3c\x4f\x86\xf4\xd7\xa5\x2c\x0a\x23\x73\x2a\xb5\x72\x24\x57\xe0\x51\x3c\xf0\x47\x50\x6d\xc2\x3c\x8d\x18\x4f\x23\xdc\xb1\xbd\xa5\x2f\xe4\x3f\x8e\xff\xe9\xc2\x36\x66\x61\x30\x60\xbe\x7f\x25\x20\x55\x46\x7f\x6a\x0f\x70\x22\x0d\x73\xb3\xd4\x23\x2b\x7d\x72\x2a\xf6\x5d\xf9\x45\x54\xef\x7d\x6b\xe1\x8a\x26\xd4\x0c\xb8\xb3\x34\x10\xf8\x7f\xab\xfa\xe0\xec\x5c\x51\x55\xbd\x6d\x49\xaa\x81\x40\xf8\xad\x34\x38\xca\xcf\x8b\x98\x17\xeb\xd5\x39\x94\xc4\xca\x2d\x7d\x71\x83\xfb\x39\x26\x28\x87\x95\xbd\x60\x7c\x0b\x34\xd7\xd1\x0a\x7f\x31\x45\xf9\x2d\xba\x30\x7b\x5e\x9a\x1e\x39\x10\xd6\xcb\xbc\xa5\xfc\x19\xab\x39\x78\x67\x01\x41\x07\xb7\x7e\x4a\x1d\xf6\xbd\x42\x55\xda\x88\xc2\xfb\x99\xf6\xfe\x99\x8f\xa3\x47\x1d\x6b\x64\x8d\x01\x9d\xc5\x98\x1c\xa4\xe4\xa3\xcb\x1e\xc8\x62\xf6\xf7\x9e\xc8\x3d\x4e\x1e\xdf\x35\x85\xb2\x35\x9f\xdb\x06\xfa\x2b\xdb\x7b\x38\xdc\x5e\xc6\xd5\x4a\xa6\xd9\x3a\x03\x7f\x38\x4e\xf5\x87\x8c\x05\x73\x23\xba\x92\x9f\xfa\x8f\x12\xa5\x74\xc6\x6d\x28\xf7\xd6\x8f\xad\x3a\x9f\x0c\x09\x9a\x4f\x8f\x99\x59\xfb\x8b\xef\x39\xe1\x29\xe2\x1e\x0a\x9b\x66\xea\x86\xf2\xae\x19\x9f\x30\x85\x2b\x7b\x3a\xcb\x22\xee\xc0\x6b\x58\x0e\x98\x11\x0a\xfd\x0d\x05\xc1\x8e\x43\xba\x76\x4e\xf0\x95\xff\x60\xc1\xa5\x15\x2a\x33\x72\x56\x9b\xfa\x9f\xbb\x05\x3e\x77\xf8\xe7\xe4\xa6\xd5\x0f\xc9\xd2\xea\xf5\x40\x83\x54\x22\x0b\x04\x2c\xf9\xfa\x4f\xeb\xc0\x5a\xf6\xc6\x90\xce\x3f\x48\xf7\xb8\x69\x0a\xb4\x97\x46\x28\xf3\x92\x76\x15\x51\xd7\x32\x21\x34\xff\x5f\x25\xb6\xde\x0d\xb9\xd2\x42\x13\xc7\x56\x1e\x17\xfa\xc6\x89\x37\x17\x10\xff\x87\xb1\xee\x4b\x43\xa0\x38\xc0\xca\x69\x5b\xf2\x72\x0e\x69\x01\x09\xad\xd4\x5e\xf1\x65\xd5\xe6\x61\x71\x52\xd9\x2b\x2f\xbe\x21\xfb\x14\x0e\x40\x38\xc2\xa7\xe6\x63\x75\x7a\xcd\xe0\x4b\x1c\x43\x2e\x1a\x8e\xea\x53\x1a\x26\x96\x95\x9c\xbe\x25\x1e\x9a\xe6\xf4\x22\x08\xb9\xc2\xa4\x80\xab\xd1\x54\xa0\x46\xc3\x41\x48\xb8\x96\x68\xa7\x08\xc7\x6e\xe6\xc1\x48\x29\xf4\xe9\x70\x58\x31\x79\xbd\x19\x68\x9b\x1d\x40\xa9\xc0\x8b\xcb\x83\xbc\x93\x10\x4f\x9f\xfb\xec\x82\x14\xb1\xc9\x19\x2e\xb3\x3b\x4a\x6d\xee\x02\xb6\x48\x43\xca\x14\xec\xcf\x46\xf3\x03\x75\xa2\xa2\x1b\x1e\x6e\x02\xbc\xd8\x19\x6f\xb5\x1f\x9a\xe7\x00\x33\xe0\x68\x56\xa5\xf7\x9d\x81\xf5\xa1\x0a\xa8\x80\x31\x51\xcb\x5f\x31\xb1\xf2\x5b\x91\x9b\x98\xac\x66\x10\x06\xc5\x27\xd9\x1b\xef\xa9\x5f\xba\x98\x7c\xf3\x77\xd5\x7b\x27\x98\xf3\xf6\x68\x6a\x92\x99\x22\xf1\xb6\x45\x6f\x83\x44\xee\x30\x0a\x39\x72\xcb\x28\x51\x9d\x93\x09\x10\xf8\xec\xa8\x36\xb1\x1a\x2b\x1a\x8c\x9e\x8d\x2d\x5b\xa0\x1b\x97\x27\x65\x00\x30\x14\xb3\x87\x75\x1d\x72\xb2\x0c\xdc\xa6\xc8\x04\xac\xd8\xa0\xb7\xc5\x46\x56\xa2\x82\xc4\x92\x1f\x11\x88\xa5\x87\x34\x4c\x41\xf1\xd0\x0b\xc5\xcd\xea\xf6\x8f\xc6\x3b\x65\xd1\xa6\x4b\x9b\xd3\xe8\x6a\x96\x87\xb5\xac\x64\xcd\x6d\x6f\x4d\xeb\x73\x6e\xf7\x05\x6c\x59\xfc\x8d\xba\x05\x6f\x75\xca\x23\xf0\x76\xed\x4f\x30\x7c\xe3\xbf\x0c\xad\xee\x89\x42\x0e\xec\xaa\x3f\x8a\xe7\xc9\x94\xa9\x68\x39\x2a\x15\xf2\x21\xcc\xf0\xed\x5c\xa9\xed\x72\x8f\xf7\x6a\x12\x66\x86\x0d\x89\x56\xdc\x24\x78\xe6\x5a\x99\xed\x40\xff\x9c\x9d\x28\xde\xe7\xd3\x59\x4e\xe5\x57\x1b\x23\xaf\xe9\xe1\xd4\x58\xfd\xc9\x1f\xaa\xaf\x23\xa3\xc4\x54\xb0\xbb\x48\xa8\x80\xd9\xdd\x1d\x1d\xd9\x89\x28\xa5\xa8\x26\x4c\xfc\x94\xa4\x1b\xde\x19\xe6\xb0\xd1\x4b\xf5\x47\xed\x28\xd3\x81\x63\xe8\x9f\x52\x26\x32\x75\xe0\x46\xb2\x14\x02\x48\x6d\xfd\x4e\x3f\xb2\x91\x6c\xa9\xa8\xbe\xf5\xc8\x6d\x96\x31\xf6\xba\x80\xf2\x15\x6b\xee\x41\xfe\x19\xfb\x39\xd2\x4d\x4b\x3f\x67\x20\xbc\xe3\x4d\x57\xc4\x4c\xb6\x7c\x57\x5b\x12\x1e\x75\x7c\xb3\x67\x59\x66\xea\x59\x11\x96\x31\x06\xd7\xdd\xc4\x2f\x29\x1f\x1c\x3a\xa7\x37\xe9\x7b\x21\xe4\x27\xe9\x32\x44\xa9\xc6\xd7\xb6\x39\xae\x61\xa3\x5a\x65\x25\xae\xb9\x0d\x80\x3f\xe7\xdf\xc1\xcc\x6b\x28\x55\xb8\xe6\xcf\x92\x2a\x5b\x3d\x22\x43\xd8\x84\x15\x07\xee\xa3\x4a\x27\x78\x5a\x26\x86\x39\xa9\x30\x52\x26\xe4\xbc\xb8\xd6\xc3\xfa\xc4\xb6\xa7\xa7\xa2\x92\x04\x2f\x55\x6b\x4c\x28\x28\x28\xb6\x00\xe9\x20\x3d\x4a\xb7\xa3\xa4\x00\x45\x9b\xcb\xa2\x1b\xad\x75\xbc\x67\xd1\x0a\xcf\x9a\x43\x43\x15\xde\x05\xdf\x03\x11\xcd\x22\x8e\xa7\x03\xeb\xf1\x72\xc4\xad\xf0\x9a\x34\x29\x28\xd1\x79\x3c\x98\xc2\xeb\xe2\x45\x35\xb9\x22\xfe\x18\xff\x5b\x56\x4a\xe6\x41\x3a\x71\x12\x9b\x45\x2c\x9e\x08\x58\xcb\xe4\x22\xaf\x4c\xb5\x43\xae\x08\x7a\x55\x58\xf8\xd8\x50\x20\x13\x1e\x19\x8e\x11\x81\xba\x55\x08\x6c\x58\xff\xc1\x28\x8f\xd8\xf8\x4d\x36\xac\x55\xfa\x11\x25\x9f\x71\x77\x59\x6b\x58\x70\x52\xb3\x56\xc0\xfd\x57\x32\xc0\xc1\xe8\x34\xd7\x11\x58\x9e\x4c\x84\x81\x67\x2e\x02\xfa\xc7\xdf\x52\x8c\x9f\x17\x2d\xfb\x80\x0b\xcb\x45\x17\xca\xcc\xe2\x94\x95\x0f\x71\x79\xf0\x1c\xe7\xb5\x01\x7f\xe4\x83\x29\xcc\xc8\xac\x08\x51\x3c\x66\x29\x97\x0e\x1a\x68\xb2\x89\x13\xaa\xb3\xfc\x8f\x7e\xc9\x64\xae\x34\xe8\x2d\x42\xb3\x87\x72\xdc\x6e\xd0\xf4\xea\x6e\x87\x88\x76\xc4\x57\xa1\xae\xaa\x48\x56\x3c\x0e\x77\x0a\x8b\x5f\xe6\x18\x94\x52\x89\x74\xb5\xee\xcf\x9f\x3e\x51\x1e\x61\xb4\xce\xef\xaf\xcb\xb0\x3c\x1d\x7b\x70\x5f\x68\xcf\x4b\x36\xa3\xa8\x76\xb4\x93\x44\xdd\x12\x5a\x97\xee\xfb\x32\x6a\x03\xa9\x32\x55\xb4\x27\x60\x47\x48\x50\x40\x57\x2a\x2c\x1a\x64\x79\x5f\x81\x2e\x5a\x8d\xf4\x20\x91\xbe\x54\xfb\xc3\x7f\x86\xd1\x91\x10\x44\x38\x9e\xca\x81\x7f\xdf\x58\xae\xda\xbc\x63\x2c\xbe\x95\x4b\x24\xdd\x2e\xac\xa0\x91\xa9\x3b\x42\x44\xd1\x9c\x74\xce\xf9\x98\x22\x5b\x12\x41\x27\x82\xc4\x56\xeb\x2c\x85\x95\x70\x65\x3f\x63\x7e\xb8\xa5\x0f\xe5\xbf\xaa\xc6\x01\xdf\xbc\x7d\x71\x00\x22\x05\x10\x49\xa8\x8a\x51\xe9\x22\x47\x2e\xd7\x83\xdb\x01\x95\x38\xe5\xe7\x8b\x76\x77\x47\x09\x31\x33\xbe\x47\x36\xe8\x5b\x0f\x47\xe8\x58\xc9\x23\xe6\xd4\x57\xfb\xc6\x09\x5b\x9b\xdd\xec\x14\x47\x2a\x39\xeb\xdf\x56\xea\xd6\xc1\xeb\xda\x68\x04\xd8\x01\xdf\x13\x52\x21\xb9\xbe\x26\xb8\x0a\xaf\x58\x3f\x6f\xd2\x9a\x40\x6b\x1b\xfd\x7e\x01\xbb\x04\x14\x6e\x8a\x89\x7f\x97\x5b\xc4\xb6\xcf\x82\x6f\x00\x8c\x96\x20\x26\xef\x6e\xa5\xd5\x26\x2c\x8b\x5e\xd0\xb6\x59\x2a\xca\xbb\xa6\xeb\xea\x81\xad\x65\xc0\xb7\xed\xed\x99\xb3\x9c\x29\x15\x91\xcd\xab\x2e\xe5\x67\x0a\x76\x3c\x97\x32\x3e\x42\x94\x47\x9a\xc1\x21\x4b\xa2\x09\xde\xd4\xd2\xaf\xab\x4e\xd8\x78\xa6\xf2\x31\xe4\xe5\xea\x7f\x56\x36\x11\x75\x17\x6f\x36\x17\xfa\x49\x18\x89\x0e\x04\x93\xc3\x98\x77\x71\x1b\x7f\xcc\x00\x78\x90\x81\x6e\xf5\x5e\x50\x5a\xa7\x36\xbc\x39\xfd\x28\xdd\xfd\xea\x64\xee\x7b\x6b\xc8\x4b\x96\x53\x57\x0e\xf9\x3c\xb6\x39\xe7\xea\xce\x94\xfe\x49\x28\xfb\x94\x33\x43\x92\xd7\x3b\x40\x98\x58\x23\x90\xdc\x8d\x9d\x26\x84\xf8\x3d\xb5\xb9\x97\xdb\x46\x1e\x07\x09\xee\xe9\x18\x0d\x37\xdb\x7f\x50\x5a\x64\xf6\x72\x94\xb8\xfc\x4d\x88\xe8\xb6\x05\xd3\x28\xaf\x55\x8a\x1e\x16\x10\x65\xbd\x39\x5c\x43\xd1\x6b\x9c\xd6\x70\x5e\xc4\xd9\xe2\x6f\x99\xe3\x70\x24\x7e\x97\xad\x04\x9b\xc4\x24\x14\xb5\x98\xbc\x21\x03\x76\x1f\x10\xe6\xe0\x6b\x28\x58\x67\x83\xdf\xdd\x80\x6e\xb7\x65\xf3\x8a\xee\x95\x25\x1b\x6f\xa1\xbb\x60\x3f\xdd\x01\xb7\xb7\x76\xe2\xb2\x3a\x5b\xcf\xd5\xe3\x90\x7f\x2e\x72\xdd\x3a\x2f\x0c\x35\x6c\xae\xeb\x80\xd5\xa1\x6c\x20\x51\x05\xc4\xb4\x0d\xa5\x17\xc5\xa6\x21\x6d\x4b\x6e\xbc\xb6\x02\x44\xce\x75\xb4\x9e\xd6\xc8\xeb\xf1\x5e\xc0\xe4\x8e\xae\x79\x5f\xc0\xe5\x0d\x4d\xff\x69\xf3\xc5\x64\x4a\x9e\xf6\x87\x02\x45\xe7\xb6\xe1\x00\xb4\x33\x25\xfa\x4c\x5a\x87\xb1\x8f\xc5\x97\xc4\xb6\xce\xf9\x35\x5d\x84\xe2\xbb\x12\x23\xce\x04\x9f\xee\x87\x38\x67\x4b\x5b\x52\x6b\x47\x63\x4d\x09\xa3\x13\x50\xf2\xf1\x7d\x2d\x1b\x93\x32\x18\x83\xd9\x8e\xf9\x20\x0f\x8c\x3e\x35\xe4\x10\xc2\x75\x5a\xd6\xfa\x2b\x5b\x14\x4b\x99\xa7\xfc\x53\x20\xac\xeb\x49\x74\x82\xde\xe0\x35\xc1\xe4\xd2\xc7\x4c\x42\xab\xdb\x38\x84\x86\x45\x36\x77\xa1\x3f\x73\x98\xb6\x35\xc9\xd4\x33\xf5\xd7\xf8\x18\x38\xe2\xf8\xff\x5b\x34\xd9\xac\x93\xb5\x1c\xbf\xdd\xe9\x92\x53\x26\xc4\xd2\xd2\x0e\xbd\x37\xe8\xa8\x39\x80\xb8\x61\x70\xd6\xe7\x44\xab\x9d\x91\x53\xcf\x1d\x64\xc5\xf5\xe4\x16\xda\x54\x82\x29\x51\x02\x20\xa9\x20\xd6\x07\xcc\x23\x97\x82\xd4\xf4\x65\x5b\x9d\x25\x24\x8b\xae\xb4\x74\x85\x6e\x8e\x89\x2b\xb4\x1b\x85\xea\x8a\x01\xa9\x24\x15\x31\x57\xf0\xae\x3d\x7c\xca\x83\xc9\x7a\xd7\x69\x93\xae\x15\x86\xc5\x93\xa8\xda\xf3\x0d\x80\x52\x24\x5d\xe1\xf1\x17\xfd\xdc\x64\x14\x81\x5f\xf5\x82\xb3\x69\x7a\xc3\xd6\x24\xef\xec\x2d\x28\x53\xb9\xf8\xc8\xdb\xc8\xbe\xe0\xe5\x51\x72\x47\x9e\x78\xe8\xcc\x35\xb1\x61\x32\xb8\x51\x4b\x41\x31\x26\x68\xfb\xda\x09\x8d\x1d\xe9\xcc\x94\xed\x3a\xd7\x3f\x76\x07\xa3\x3b\x89\x16\xef\x9d\xd9\x07\xe0\x66\xc9\x8f\xbf\x19\x30\xe1\xfc\xfe\x73\x97\xea\x0d\xbb\xeb\x29\xba\xb1\xa7\x93\x0a\x10\xff\xc9\x05\x7a\xbb\xc4\x74\x2a\x83\x26\xc4\xc9\x41\x82\xc0\xbd\x41\x91\x48\xe7\xee\x35\x43\x5f\x2f\x8b\xe3\x6c\x3f\x99\x9d\xd5\x9f\x33\x9a\x21\x01\xd5\x4c\x2e\xef\xd8\x52\x91\x7a\x06\x8f\xe3\x2b\xfc\xeb\xa0\x42\xd0\xfd\x02\xd0\xa9\xd5\x84\xf3\x22\x20\xb1\xb6\x83\x9f\x7d\x10\x10\x55\x62\x5d\xcf\x01\x54\x65\x95\xe5\x50\xb5\x03\xe1\x1c\x2b\x7f\x3a\xc7\x48\x01\x13\xc5\x73\x2c\x8e\xe5\xaf\x4f\x1c\x4e\x21\x0b\x22\x6e\xcb\xb8\xcf\x54\x6a\x8d\x56\xba\x11\xa3\x2b\x76\x4e\xdb\xef\x27\x4e\x15\x1f\xc6\xa1\x2c\x33\xc3\x36\x5c\xaa\x3e\x2d\xf0\x71\x9e\x1c\xa2\xf3\x07\x08\xcb\x01\x9b\xcd\x22\x1f\x17\x07\x28\xf6\x0a\x63\xd0\x5b\xba\xea\x12\x80\x0a\x9a\x1b\x01\x48\x43\x6a\x01\xcb\xbe\xca\x34\x15\x7b\x13\xbe\xee\x2a\xd2\x85\x2a\x03\x51\x3b\x9b\x22\x32\x1a\xa5\x30\xec\xdf\x70\x9d\x77\x4c\x5c\x90\x41\x43\x3c\x01\xe6\x77\x48\x2f\x46\x41\x02\x3d\x44\x01\xe7\x42\x08\xb1\xf3\x95\x96\x90\x67\xcb\x4d\xda\x7f\x70\x82\xe9\x66\x4a\xe0\x4a\xe7\x21\x07\x10\xec\xf6\xa2\x79\xb1\xd6\xa3\x2a\x7d\xe9\x09\x8a\x30\x15\x1f\xdb\x70\x4a\xa3\xac\xcf\xaf\xa7\x19\xde\x98\x68\xa5\x9f\x24\xa7\x3a\xf5\xf0\xc7\x8d\x31\x05\xc3\x23\x8a\x7e\x7d\xce\x47\x76\x87\xaf\xb0\xc2\x97\x5e\x78\xdc\x90\xf7\xc2\x4d\x8d\x77\x87\x29\x14\x81\x06\xd0\x7b\x17\xa5\xfd\xe0\x8e\x04\xee\x81\x69\x72\xd1\x3e\xe9\xe5\x76\x83\xa2\x19\x42\x55\xb0\xf5\x3e\xdb\xc0\xda\x41\x67\xc1\x4a\x29\xae\x69\xcd\xf8\xea\x1b\xfc\x25\xd7\x98\x0f\x3b\xb1\x23\xe7\xce\x29\xc4\x86\xd0\x67\x0d\x8d\x84\xf5\x68\xc4\xa6\xfc\x77\x1e\xdf\x10\xb5\x96\x84\x0e\xc2\xb0\x8a\xcb\xf5\xc3\x6b\x13\x1b\x59\x9a\x46\x43\x88\xc1\xe6\x3a\x02\x50\xf5\xbc\x6c\x88\x9f\x81\x4f\x7a\xa4\xfe\x34\xdd\x18\xf2\x69\x85\xe2\xd1\xbd\x0b\x3a\xc3\xfd\xc5\x8c\xdc\xdf\x72\x49\xa5\x7b\x9e\x5e\x49\xb4\xf1\x34\x04\x92\xac\x1b\x18\xf1\xfc\x6f\x2c\xa2\xf2\x0c\x7b\x25\xfe\x02\x77\x03\xef\x8e\x38\xf2\xdc\x29\xd0\x78\x6e\xc5\xbb\x16\xc2\xc8\xca\xd9\x68\xfb\x41\x3a\x4d\x77\x91\x58\x93\x96\xdd\xce\x6b\x2d\x12\xcf\x52\xd8\x59\x89\x05\x32\x7b\xeb\xdb\xfd\x86\x5a\xde\xb4\x40\xd6\xe5\x23\xc9\xd3\x2c\x46\xbc\xef\xe4\xb8\x54\x4c\xad\x6a\xf3\xda\x31\xe3\x29\xe1\xe3\xa5\xa0\x1f\x90\xdc\xe8\x62\xfe\xcb\xb1\x0a\xaa\x9e\x48\xb1\x6c\x9a\x97\xad\xc0\xf3\x6e\xec\xc6\x07\xfc\x00\x54\x70\xdb\xbd\xe5\x24\x33\xa3\x18\xd1\xf9\x0e\x2a\x74\xad\x1b\xeb\x91\x66\x55\xe7\xf5\xf2\xef\x45\x45\x9e\x7a\xcb\x43\xa3\x10\x42\xa1\x1f\x6d\x17\x16\x98\x53\xc3\xb0\x70\x29\x1e\xdb\x99\x13\xfe\x77\x12\x9d\x2c\x94\xfa\x56\x58\xf5\x27\x5f\x52\x40\xeb\x2b\x36\x89\xc9\xc6\x12\x88\x04\x0b\x64\xa1\x35\x53\xf6\x1c\x95\xcc\x36\x97\xce\xd6\x6d\x62\x1c\x51\x1d\xc9\x8f\xc4\xc3\x9d\xaf\xfa\xcd\x0e\x3b\xeb\xc0\xf6\xd5\x24\x84\xd2\xde\x51\x58\xfd\x46\x97\x70\xc3\x94\x5a\x6b\x18\x01\xb1\x8a\x19\x54\x2a\xd8\xd9\x0b\xab\xd2\x37\xfe\xca\x77\xa6\xc7\x08\x31\x76\x5b\xe6\x14\x0f\x16\xbc\xe7\xf6\x1b\x96\xe4\x55\x62\x56\xd0\x9d\x1e\xa6\xd8\x10\x26\x22\x02\x1d\x07\xa1\xd7\xaa\x87\x86\xbd\x7c\x45\x29\xc8\x7c\x91\x35\x6d\x2d\xc9\x3c\x9b\x59\x2d\x0c\xec\xf4\xc0\x0d\xf5\xb2\xdf\x06\xb2\x23\x4e\xb0\x5e\x00\x67\xfa\xed\xd6\x19\x73\xa6\xe5\x61\x73\x26\xce\x24\x13\xbb\xbf\xfe\xc3\x72\x8d\x26\xae\x12\x26\x80\xd1\x27\x2a\x04\x34\x60\x67\xe8\x3c\x2b\xa2\xbd\xb4\x20\xb0\x10\x4f\x89\xf7\xd3\xdc\x9e\x6d\x49\x88\xbb\x5f\x2a\xbf\x79\x7f\x52\x35\x23\x48\x8e\x0f\xb7\xb2\xc1\xc2\xe6\x43\x2e\x7b\x35\x83\x9d\x5f\xde\x2c\x6d\xff\xa6\xb1\x01\x8b\x38\x1c\x56\xbb\x07\xe3\xb8\x90\xde\x08\x80\x30\x50\x3f\x66\x0c\x37\xb8\x34\x52\xa2\x37\x68\x3f\xce\xbe\x59\x3e\x87\x37\xba\x40\xfb\xaa\x28\x28\x54\x66\x7b\xc0\x32\x04\x9d\x7d\x81\x7c\xc7\xd2\x13\x07\x38\xf4\x7b\x58\x86\x4f\x22\x4c\x9b\xfe\xc2\x45\xf8\xf0\x1d\xb7\x21\xcc\x09\x55\x34\x27\x5c\xae\xf5\x15\xe6\x3b\xcf\x13\x0f\x5f\xce\xfb\xb2\x96\xe1\x63\xa3\x74\x74\x78\x42\x39\x95\x16\x7e\x31\xa9\x01\x95\x63\x20\x10\xd7\xcd\xa3\x92\x83\x9f\x54\x16\x4a\x06\x79\x80\x73\xd2\x55\x90\x7f\x58\x83\xcf\x77\xcf\xec\x9d\x5c\x2f\x5f\x8e\xe7\x02\x5a\xd7\x94\x05\xe8\x78\x76\x73\x94\xe5\xed\xd9\xbf\x4f\x82\xbb\x08\x0d\xa4\x6f\x71\xb6\xc5\x5b\xee\x65\x26\x4b\xf7\xc2\xdf\x08\x44\xa1\x1d\x8a\x7b\x1c\xb0\x82\xa7\x6a\xdd\x35\xad\xb3\x6f\x1a\xf2\xd2\x80\x59\x3b\x6f\xe6\xcc\x7f\x53\xed\x19\xc3\x68\xaa\x19\xe5\xa1\xcb\xca\xca\x69\x2a\x13\x08\x85\xdc\xcd\xe2\x90\x66\xae\xf7\x2b\xd6\xbb\xef\x9f\x14\x02\x5a\xbd\x92\x38\x37\x87\x65\xde\x82\xf8\xac\xf4\x03\xf9\x23\x51\x3a\xdd\xec\xc8\xe9\x2c\xb6\x84\x37\x4b\xff\xf9\xfd\xb1\x49\xdf\x3b\x95\xcf\x1f\x5a\x5b\x6b\x6a\x0d\xa3\xd2\xe4\x36\x41\x76\x07\x4c\xed\x72\x0f\xd7\x32\x0b\x1e\x04\xff\xdf\x1f\xb0\x4e\x25\x2c\x76\xda\x3c\xe5\xfd\xfc\x29\x13\xc1\x6c\x8f\x4c\x0e\x8a\x01\x74\xab\x1f\xf8\x2e\x74\xdb\x57\xaf\x87\x7a\x72\xe7\xf0\xb4\xc9\x14\xf3\x62\x15\xba\xaa\x95\xcd\x1f\x09\xbf\xc2\x39\x11\x21\xdc\xe4\xee\x3b\x91\x7c\xf6\x3f\xa4\x22\x43\x39\x90\x3d\xa4\xa4\xb8\xd5\x97\x4b\xdd\xac\x9b\xba\x16\xe0\xba\xef\xa1\x21\xd0\xb8\x6b\x89\x57\x54\x94\xf2\x32\xa8\xde\xf3\x58\xe0\x6d\xee\xb7\x85\x92\xfe\x66\x1a\x7d\xad\x09\x49\x6e\xbc\x89\x2b\x3c\x5a\xaa\x76\x30\xf2\x0a\x0c\x92\xe8\x7d\x7b\x49\xc9\xba\xf2\x9f\xde\x9a\x45\x8c\xe2\x31\xf5\x55\x3d\x9e\xb2\x3e\xf4\xdc\x46\x97\x15\xcb\x9b\x2b\xbf\xba\x26\x15\x54\x2b\x8d\x32\x9d\x4b\x14\x02\xf7\x3b\xea\x60\x46\xf9\xe3\x64\x59\x08\x64\xe7\xe8\x5b\x34\x8a\x12\xcb\xa0\x9f\x3e\x26\x7a\x63\x51\x07\x6b\x1c\x47\xb0\x4a\x1b\xaa\xe0\xdd\x3f\x45\xe2\x09\x47\x51\xf7\xf6', 2)
  File "<frozen iva.efficientdet.scripts.train>", line 141, in <module>
  File "<frozen iva.efficientdet.scripts.train>", line 71, in main
  File "<frozen iva.efficientdet.scripts.train>", line 91, in run_executer
  File "<frozen iva.efficientdet.executer.distributed_executer>", line 349, in train_and_eval
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1191, in _train_model_default
    features, labels, ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "<frozen iva.efficientdet.models.det_model_fn>", line 657, in efficientdet_model_fn
  File "<frozen iva.efficientdet.models.det_model_fn>", line 547, in _model_fn
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/experimental/loss_scale_optimizer.py", line 123, in compute_gradients
    grad_loss=grad_loss)
  File "/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py", line 496, in compute_gradients
    gradients = self._optimizer.compute_gradients(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/optimizer.py", line 537, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
    unconnected_gradients)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gradients_util.py", line 694, in _GradientsHelper
    out_grads[i] = control_flow_state.ZerosLikeOutsideLoop(op, i)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_state.py", line 805, in ZerosLikeOutsideLoop
    return array_ops.zeros(zeros_shape, dtype=val.dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 2350, in zeros
    output = fill(shape, constant(zero, dtype=dtype), name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 171, in fill
    result = gen_array_ops.fill(dims, value, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 3602, in fill
    "Fill", dims=dims, value=value, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[53282,1],0]
  Exit code:    1
--------------------------------------------------------------------------
Telemetry data couldn't be sent, but the command ran successfully.
[WARNING]: <urlopen error [Errno -2] Name or service not known>
Execution status: FAIL

Please try to train with lower batch-size.

Thank you @Morganh However. After I changed batch-size to 8 as the cell below, I got new error message: Caught signal 11 (Segmentation fault: address not mapped to object at address 0x10) in the process of training efficientdet.

How should I do to deal with the problem ?

The number of GPUs I used is 4

# Apply changes for any of the parameters listed in the previous cell as required
# Example for detectnet_v2 (for each network the parameter key might be different)
specs["training_config"]["num_epochs"] = 10# num_epochs is the parameter name for all object detection networks 80 

# for efficientdet
specs["training_config"]["train_batch_size"] = 8
specs["training_config"]["num_examples_per_epoch"] = 1965 #number of images in your dataset/number of gpu's
specs["dataset_config"]["num_classes"] = int(num_classes) #num_classes was computed during kitti_to_coco_conversion
specs["eval_config"]["eval_epoch_cycle"] = 10

if "image_extension" in specs["dataset_config"].keys():
    specs["dataset_config"]["image_extension"] = "jpg"
[GPU 00] Restoring pretrained weights (309 Tensors)
2023-04-18 06:23:26,598 [INFO] root: Pretrained weights loaded with success...

[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:75   :0:482] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x10)
==== backtrace (tid:    482) ====
 0 0x0000000000043090 killpg()  ???:0
 1 0x000000000765f118 tensorflow::BinaryOp<Eigen::GpuDevice, tensorflow::functor::mul<float> >::Compute()  ???:0
 2 0x00000000010df3db tensorflow::BaseGPUDevice::Compute()  ???:0
 3 0x000000000113caa7 tensorflow::(anonymous namespace)::ExecutorState::Process()  executor.cc:0
 4 0x000000000113d10f std::_Function_handler<void (), tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(absl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8ul, std::allocator<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode> > const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*)::{lambda()#1}>::_M_invoke()  executor.cc:0
 5 0x00000000011f1725 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop()  ???:0
 6 0x00000000011ee268 std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke()  ???:0
 7 0x00000000018d69a0 execute_native_thread_routine()  /dt9-src/libstdc++-v3/src/nonshared11/../c++11/thread.cc:80
 8 0x00000000018d69a0 std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >::~unique_ptr()  /dt9-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:292
 9 0x00000000018d69a0 execute_native_thread_routine()  /dt9-src/libstdc++-v3/src/nonshared11/../c++11/thread.cc:79
10 0x0000000000008609 start_thread()  ???:0
11 0x000000000011f133 clone()  ???:0
=================================
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] *** Process received signal ***
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] Signal: Segmentation fault (11)
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] Signal code:  (-6)
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] Failing at address: 0x4b
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f33949fe090]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 1] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_cc.so.1(_ZN10tensorflow8BinaryOpIN5Eigen9GpuDeviceENS_7functor3mulIfEEE7ComputeEPNS_15OpKernelContextE+0x1e8)[0x7f33179ba118]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 2] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(_ZN10tensorflow13BaseGPUDevice7ComputeEPNS_8OpKernelEPNS_15OpKernelContextE+0x3cb)[0x7f330f4593db]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 3] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(+0x113caa7)[0x7f330f4b6aa7]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 4] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(+0x113d10f)[0x7f330f4b710f]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 5] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(_ZN5Eigen15ThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x285)[0x7f330f56b725]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 6] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x48)[0x7f330f568268]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 7] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(+0x18d69a0)[0x7f330fc509a0]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 8] /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f33949a0609]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] [ 9] /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f3394ada133]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00075] *** End of error message ***
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:76   :0:410] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x10)
==== backtrace (tid:    410) ====
 0 0x0000000000043090 killpg()  ???:0
 1 0x000000000765f118 tensorflow::BinaryOp<Eigen::GpuDevice, tensorflow::functor::mul<float> >::Compute()  ???:0
 2 0x00000000010df3db tensorflow::BaseGPUDevice::Compute()  ???:0
 3 0x000000000113caa7 tensorflow::(anonymous namespace)::ExecutorState::Process()  executor.cc:0
 4 0x000000000113d10f std::_Function_handler<void (), tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(absl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8ul, std::allocator<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode> > const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*)::{lambda()#1}>::_M_invoke()  executor.cc:0
 5 0x00000000011f1725 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop()  ???:0
 6 0x00000000011ee268 std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke()  ???:0
 7 0x00000000018d69a0 execute_native_thread_routine()  /dt9-src/libstdc++-v3/src/nonshared11/../c++11/thread.cc:80
 8 0x00000000018d69a0 std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >::~unique_ptr()  /dt9-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:292
 9 0x00000000018d69a0 execute_native_thread_routine()  /dt9-src/libstdc++-v3/src/nonshared11/../c++11/thread.cc:79
10 0x0000000000008609 start_thread()  ???:0
11 0x000000000011f133 clone()  ???:0
=================================
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] *** Process received signal ***
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] Signal: Segmentation fault (11)
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] Signal code:  (-6)
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] Failing at address: 0x4c
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fa65c51c090]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 1] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_cc.so.1(_ZN10tensorflow8BinaryOpIN5Eigen9GpuDeviceENS_7functor3mulIfEEE7ComputeEPNS_15OpKernelContextE+0x1e8)[0x7fa5df4d8118]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 2] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(_ZN10tensorflow13BaseGPUDevice7ComputeEPNS_8OpKernelEPNS_15OpKernelContextE+0x3cb)[0x7fa5d6f773db]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 3] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(+0x113caa7)[0x7fa5d6fd4aa7]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 4] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(+0x113d10f)[0x7fa5d6fd510f]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 5] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(_ZN5Eigen15ThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x285)[0x7fa5d7089725]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 6] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x48)[0x7fa5d7086268]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 7] /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/../libtensorflow_framework.so.1(+0x18d69a0)[0x7fa5d776e9a0]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 8] /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7fa65c4be609]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] [ 9] /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fa65c5f8133]
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:00076] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w:78   :0:427] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x10)
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node c482f0b6-a8f7-4418-9b14-ea4b1e75a776-h9b4w exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
Telemetry data couldn't be sent, but the command ran successfully.
[WARNING]: <urlopen error [Errno -2] Name or service not known>
Execution status: FAIL

To narrow down, please try to run below experiments.
1st experiment: set specs[“training_config”][“train_batch_size”] = 1

2nd experiment: run with 1gpu

3rd experiment: run with

Can efficientdet not run with multiple GPUs and be trained by AutoML ?

Efficientdet can run with multiple GPUs. And AutoML also supports EfficientDet.

It succeeded when I ran with 1 gpu.

However. Why the error occurred when I ran with multiple gpus ?

Recently, did you run multiple gpus successfully with other network?
And how about $nvidia-smi ?

I found that it also succeeded when I trained efficientdet model with 4 gpus by setting specs[“training_config”][“train_batch_size”] = 1 first.

It seemed like that the value of specs[“training_config”][“train_batch_size”] cannot be above 8 ?

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

There is not limitation for “train_batch_size” as long as there is enough GPU memory.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.