Not able to run deepstream-ssd-parser python example

Please provide complete information as applicable to your setup.

• Hardware Platform (GPU) RTX 3050 Ti
• DeepStream Version 5.1
• TensorRT Version The one that came installed with deepstream 5.1 triton image
• NVIDIA GPU Driver Version (valid for GPU only) 465.27
• Issue Type( questions, new requirements, bugs) question

Hello, I’m unable to run deepstream-ssd-parser python example.
I prepared the model using ./prepare_ds_trtis_model_repo.sh
and then used the sample video /opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_1080p_h264.mp4

The pipeline buids, and then it stops after compiling for a bit, i’ve attached the logs here with

2021-11-16 04:05:07.155472: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.31GiB (rounded to 2477328384). Current allocation summary follows.
2021-11-16 04:05:07.155573: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256): Total Chunks: 140, Chunks in use: 140. 35.0KiB allocated for chunks. 35.0KiB in use in bin. 4.8KiB client-requested in use in bin.
2021-11-16 04:05:07.155594: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512): Total Chunks: 49, Chunks in use: 49. 28.8KiB allocated for chunks. 28.8KiB in use in bin. 26.0KiB client-requested in use in bin.
2021-11-16 04:05:07.155617: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024): Total Chunks: 18, Chunks in use: 17. 20.2KiB allocated for chunks. 19.2KiB in use in bin. 17.9KiB client-requested in use in bin.
2021-11-16 04:05:07.155643: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048): Total Chunks: 6, Chunks in use: 6. 13.2KiB allocated for chunks. 13.2KiB in use in bin. 12.7KiB client-requested in use in bin.
2021-11-16 04:05:07.155667: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096): Total Chunks: 3, Chunks in use: 3. 15.5KiB allocated for chunks. 15.5KiB in use in bin. 15.3KiB client-requested in use in bin.
2021-11-16 04:05:07.155689: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 8.5KiB client-requested in use in bin.
2021-11-16 04:05:07.155711: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384): Total Chunks: 3, Chunks in use: 3. 70.0KiB allocated for chunks. 70.0KiB in use in bin. 70.0KiB client-requested in use in bin.
2021-11-16 04:05:07.155740: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768): Total Chunks: 3, Chunks in use: 3. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2021-11-16 04:05:07.155771: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536): Total Chunks: 8, Chunks in use: 8. 575.8KiB allocated for chunks. 575.8KiB in use in bin. 575.7KiB client-requested in use in bin.
2021-11-16 04:05:07.155797: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072): Total Chunks: 18, Chunks in use: 18. 3.54MiB allocated for chunks. 3.54MiB in use in bin. 3.54MiB client-requested in use in bin.
2021-11-16 04:05:07.155821: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144): Total Chunks: 21, Chunks in use: 21. 7.32MiB allocated for chunks. 7.32MiB in use in bin. 7.32MiB client-requested in use in bin.
2021-11-16 04:05:07.155847: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288): Total Chunks: 15, Chunks in use: 15. 10.52MiB allocated for chunks. 10.52MiB in use in bin. 10.52MiB client-requested in use in bin.
2021-11-16 04:05:07.155875: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576): Total Chunks: 14, Chunks in use: 14. 19.03MiB allocated for chunks. 19.03MiB in use in bin. 18.50MiB client-requested in use in bin.
2021-11-16 04:05:07.155902: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152): Total Chunks: 5, Chunks in use: 5. 11.21MiB allocated for chunks. 11.21MiB in use in bin. 10.43MiB client-requested in use in bin.
2021-11-16 04:05:07.155928: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304): Total Chunks: 5, Chunks in use: 4. 23.78MiB allocated for chunks. 19.50MiB in use in bin. 19.50MiB client-requested in use in bin.
2021-11-16 04:05:07.155956: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608): Total Chunks: 1, Chunks in use: 1. 9.60MiB allocated for chunks. 9.60MiB in use in bin. 9.60MiB client-requested in use in bin.
2021-11-16 04:05:07.155985: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216): Total Chunks: 3, Chunks in use: 3. 59.00MiB allocated for chunks. 59.00MiB in use in bin. 55.22MiB client-requested in use in bin.
2021-11-16 04:05:07.156008: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-16 04:05:07.156029: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-16 04:05:07.156056: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-16 04:05:07.156077: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 1.39GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-16 04:05:07.156099: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 2.31GiB was 256.00MiB, Chunk State:
2021-11-16 04:05:07.156130: I tensorflow/core/common_runtime/bfc_allocator.cc:891] Size: 1.39GiB | Requested Size: 71.20MiB | in_use: 0 | bin_num: 20, prev: Size: 16.83MiB | Requested Size: 16.83MiB | in_use: 1 | bin_num: -1
2021-11-16 04:05:07.156148: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1640313600
2021-11-16 04:05:07.156170: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ca000000 next 1 of size 4864
2021-11-16 04:05:07.156204: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ca001300 next 2 of size 5660928
2021-11-16 04:05:07.156221: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ca567400 next 3 of size 1280
2021-11-16 04:05:07.156238: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ca567900 next 4 of size 2304
2021-11-16 04:05:07.156255: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ca568200 next 5 of size 10063872
2021-11-16 04:05:07.156272: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93caf01200 next 6 of size 2304
2021-11-16 04:05:07.156289: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93caf01b00 next 7 of size 5031936
2021-11-16 04:05:07.156307: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3ce300 next 8 of size 2304
2021-11-16 04:05:07.156326: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3cec00 next 9 of size 8704
2021-11-16 04:05:07.156344: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d0e00 next 10 of size 256
2021-11-16 04:05:07.156360: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d0f00 next 11 of size 256
2021-11-16 04:05:07.156378: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d1000 next 12 of size 4864
2021-11-16 04:05:07.156395: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2300 next 13 of size 256
2021-11-16 04:05:07.156413: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2400 next 14 of size 256
2021-11-16 04:05:07.156430: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2500 next 15 of size 1280
2021-11-16 04:05:07.156447: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2a00 next 16 of size 256
2021-11-16 04:05:07.156464: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2b00 next 17 of size 256
2021-11-16 04:05:07.156481: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2c00 next 18 of size 512
2021-11-16 04:05:07.156498: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2e00 next 19 of size 256
2021-11-16 04:05:07.156515: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d2f00 next 20 of size 256
2021-11-16 04:05:07.156531: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3000 next 21 of size 256
2021-11-16 04:05:07.156548: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3100 next 22 of size 256
2021-11-16 04:05:07.156564: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3200 next 23 of size 256
2021-11-16 04:05:07.156580: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3300 next 24 of size 256
2021-11-16 04:05:07.156597: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3400 next 25 of size 256
2021-11-16 04:05:07.156613: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3500 next 26 of size 256
2021-11-16 04:05:07.156629: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3600 next 27 of size 256
2021-11-16 04:05:07.156645: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3700 next 28 of size 256
2021-11-16 04:05:07.156662: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb3d3800 next 29 of size 5031936
2021-11-16 04:05:07.156679: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb8a0000 next 30 of size 2304
2021-11-16 04:05:07.156696: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cb8a0900 next 31 of size 2515968
2021-11-16 04:05:07.156713: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbb06d00 next 32 of size 2304
2021-11-16 04:05:07.156730: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbb07600 next 33 of size 248832
2021-11-16 04:05:07.156746: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbb44200 next 34 of size 256
2021-11-16 04:05:07.156762: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbb44300 next 35 of size 884736
2021-11-16 04:05:07.156779: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbc1c300 next 36 of size 256
2021-11-16 04:05:07.156796: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbc1c400 next 37 of size 442368
2021-11-16 04:05:07.156812: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbc88400 next 38 of size 256
2021-11-16 04:05:07.156830: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbc88500 next 39 of size 221184
2021-11-16 04:05:07.156847: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbcbe500 next 40 of size 256
2021-11-16 04:05:07.156864: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbcbe600 next 41 of size 221184
2021-11-16 04:05:07.156881: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbcf4600 next 42 of size 256
2021-11-16 04:05:07.156899: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbcf4700 next 43 of size 110592
2021-11-16 04:05:07.156916: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbd0f700 next 44 of size 256
2021-11-16 04:05:07.156933: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbd0f800 next 45 of size 512
2021-11-16 04:05:07.156951: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbd0fa00 next 46 of size 294912
2021-11-16 04:05:07.156967: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbd57a00 next 47 of size 256
2021-11-16 04:05:07.156984: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbd57b00 next 48 of size 65536
2021-11-16 04:05:07.157000: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbd67b00 next 49 of size 1024
2021-11-16 04:05:07.157017: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbd67f00 next 50 of size 1179648
2021-11-16 04:05:07.157034: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbe87f00 next 51 of size 512
2021-11-16 04:05:07.157051: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbe88100 next 52 of size 131072
2021-11-16 04:05:07.157067: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbea8100 next 53 of size 1024
2021-11-16 04:05:07.157084: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbea8500 next 54 of size 1179648
2021-11-16 04:05:07.157100: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbfc8500 next 55 of size 512
2021-11-16 04:05:07.157117: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cbfc8700 next 56 of size 262144
2021-11-16 04:05:07.157134: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc008700 next 57 of size 2048
2021-11-16 04:05:07.157156: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc008f00 next 58 of size 4718592
2021-11-16 04:05:07.157175: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc488f00 next 59 of size 1024
2021-11-16 04:05:07.157192: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc489300 next 60 of size 1048576
2021-11-16 04:05:07.157209: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc589300 next 61 of size 1536
2021-11-16 04:05:07.157227: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc589900 next 62 of size 1441792
2021-11-16 04:05:07.157244: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc6e9900 next 63 of size 1280
2021-11-16 04:05:07.157261: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc6e9e00 next 64 of size 2211840
2021-11-16 04:05:07.157278: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc905e00 next 65 of size 768
2021-11-16 04:05:07.157295: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc906100 next 66 of size 786432
2021-11-16 04:05:07.157312: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc9c6100 next 67 of size 1024
2021-11-16 04:05:07.157329: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cc9c6500 next 68 of size 1806336
2021-11-16 04:05:07.157345: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ccb7f500 next 69 of size 1024
2021-11-16 04:05:07.157364: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ccb7f900 next 70 of size 1548288
2021-11-16 04:05:07.157381: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cccf9900 next 71 of size 768
2021-11-16 04:05:07.157398: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cccf9c00 next 72 of size 786432
2021-11-16 04:05:07.157414: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ccdb9c00 next 73 of size 512
2021-11-16 04:05:07.157432: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ccdb9e00 next 74 of size 524288
2021-11-16 04:05:07.157448: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cce39e00 next 75 of size 1536
2021-11-16 04:05:07.157465: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cce3a400 next 76 of size 1441792
2021-11-16 04:05:07.157482: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ccf9a400 next 77 of size 1280
2021-11-16 04:05:07.157499: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ccf9a900 next 78 of size 2211840
2021-11-16 04:05:07.157516: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd1b6900 next 79 of size 768
2021-11-16 04:05:07.157533: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd1b6c00 next 80 of size 786432
2021-11-16 04:05:07.157550: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd276c00 next 81 of size 1024
2021-11-16 04:05:07.157567: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd277000 next 82 of size 1806336
2021-11-16 04:05:07.157584: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd430000 next 83 of size 1024
2021-11-16 04:05:07.157600: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd430400 next 84 of size 1290240
2021-11-16 04:05:07.157617: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd56b400 next 85 of size 768
2021-11-16 04:05:07.157635: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd56b700 next 86 of size 655360
2021-11-16 04:05:07.157651: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd60b700 next 87 of size 512
2021-11-16 04:05:07.157668: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd60b900 next 88 of size 524288
2021-11-16 04:05:07.157685: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd68b900 next 89 of size 768
2021-11-16 04:05:07.157871: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd68bc00 next 90 of size 884736
2021-11-16 04:05:07.157949: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd763c00 next 91 of size 512
2021-11-16 04:05:07.157970: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd763e00 next 92 of size 294912
2021-11-16 04:05:07.157989: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd7abe00 next 93 of size 1024
2021-11-16 04:05:07.158008: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd7ac200 next 94 of size 2359296
2021-11-16 04:05:07.158025: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd9ec200 next 95 of size 1024
2021-11-16 04:05:07.158044: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cd9ec600 next 96 of size 1769472
2021-11-16 04:05:07.158061: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdb9c600 next 97 of size 768
2021-11-16 04:05:07.158078: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdb9c900 next 98 of size 442368
2021-11-16 04:05:07.158096: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdc08900 next 99 of size 512
2021-11-16 04:05:07.158114: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdc08b00 next 100 of size 221184
2021-11-16 04:05:07.158131: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdc3eb00 next 101 of size 768
2021-11-16 04:05:07.158149: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdc3ee00 next 102 of size 884736
2021-11-16 04:05:07.158167: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdd16e00 next 103 of size 512
2021-11-16 04:05:07.158184: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdd17000 next 104 of size 294912
2021-11-16 04:05:07.158203: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdd5f000 next 105 of size 768
2021-11-16 04:05:07.158218: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdd5f300 next 106 of size 1327104
2021-11-16 04:05:07.158229: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdea3300 next 107 of size 768
2021-11-16 04:05:07.158240: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdea3600 next 108 of size 1105920
2021-11-16 04:05:07.158250: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdfb1600 next 109 of size 768
2021-11-16 04:05:07.158262: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cdfb1900 next 110 of size 368640
2021-11-16 04:05:07.158276: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce00b900 next 111 of size 512
2021-11-16 04:05:07.158292: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce00bb00 next 112 of size 221184
2021-11-16 04:05:07.158302: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce041b00 next 113 of size 768
2021-11-16 04:05:07.158314: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce041e00 next 114 of size 368640
2021-11-16 04:05:07.158324: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce09be00 next 115 of size 768
2021-11-16 04:05:07.158336: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce09c100 next 116 of size 737280
2021-11-16 04:05:07.160684: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce150100 next 117 of size 512
2021-11-16 04:05:07.160709: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce150300 next 118 of size 294912
2021-11-16 04:05:07.160719: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce198300 next 119 of size 768
2021-11-16 04:05:07.160729: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce198600 next 120 of size 921600
2021-11-16 04:05:07.160737: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce279600 next 121 of size 768
2021-11-16 04:05:07.160743: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce279900 next 122 of size 737280
2021-11-16 04:05:07.160751: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce32d900 next 123 of size 512
2021-11-16 04:05:07.160758: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce32db00 next 124 of size 294912
2021-11-16 04:05:07.160765: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce375b00 next 125 of size 512
2021-11-16 04:05:07.160772: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce375d00 next 126 of size 221184
2021-11-16 04:05:07.160780: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce3abd00 next 127 of size 768
2021-11-16 04:05:07.160789: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce3ac000 next 128 of size 442368
2021-11-16 04:05:07.160798: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce418000 next 129 of size 512
2021-11-16 04:05:07.160805: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce418200 next 130 of size 442368
2021-11-16 04:05:07.160812: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce484200 next 131 of size 512
2021-11-16 04:05:07.160822: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce484400 next 132 of size 221184
2021-11-16 04:05:07.160829: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce4ba400 next 133 of size 512
2021-11-16 04:05:07.160836: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce4ba600 next 134 of size 589824
2021-11-16 04:05:07.160843: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce54a600 next 135 of size 512
2021-11-16 04:05:07.160850: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce54a800 next 136 of size 442368
2021-11-16 04:05:07.160858: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce5b6800 next 137 of size 512
2021-11-16 04:05:07.160865: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce5b6a00 next 138 of size 221184
2021-11-16 04:05:07.160872: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce5eca00 next 139 of size 512
2021-11-16 04:05:07.160880: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce5ecc00 next 140 of size 294912
2021-11-16 04:05:07.160887: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce634c00 next 141 of size 1024
2021-11-16 04:05:07.160894: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce635000 next 142 of size 516096
2021-11-16 04:05:07.160902: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce6b3000 next 143 of size 512
2021-11-16 04:05:07.160910: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce6b3200 next 144 of size 221184
2021-11-16 04:05:07.160918: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce6e9200 next 145 of size 256
2021-11-16 04:05:07.160926: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce6e9300 next 146 of size 147456
2021-11-16 04:05:07.160934: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce70d300 next 147 of size 512
2021-11-16 04:05:07.160942: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce70d500 next 148 of size 589824
2021-11-16 04:05:07.160950: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce79d500 next 149 of size 512
2021-11-16 04:05:07.160958: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce79d700 next 150 of size 442368
2021-11-16 04:05:07.160966: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce809700 next 151 of size 512
2021-11-16 04:05:07.160974: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce809900 next 152 of size 221184
2021-11-16 04:05:07.160983: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce83f900 next 153 of size 512
2021-11-16 04:05:07.160992: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce83fb00 next 154 of size 294912
2021-11-16 04:05:07.161002: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce887b00 next 155 of size 768
2021-11-16 04:05:07.161010: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce887e00 next 156 of size 737280
2021-11-16 04:05:07.161021: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce93be00 next 157 of size 512
2021-11-16 04:05:07.161029: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce93c000 next 158 of size 163840
2021-11-16 04:05:07.161037: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce964000 next 159 of size 512
2021-11-16 04:05:07.161045: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce964200 next 160 of size 331776
2021-11-16 04:05:07.161053: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce9b5200 next 161 of size 512
2021-11-16 04:05:07.161061: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce9b5400 next 162 of size 221184
2021-11-16 04:05:07.161069: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce9eb400 next 163 of size 256
2021-11-16 04:05:07.161078: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce9eb500 next 164 of size 81920
2021-11-16 04:05:07.161086: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce9ff500 next 165 of size 256
2021-11-16 04:05:07.161094: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ce9ff600 next 166 of size 65536
2021-11-16 04:05:07.161102: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cea0f600 next 167 of size 512
2021-11-16 04:05:07.161109: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cea0f800 next 168 of size 221184
2021-11-16 04:05:07.161117: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cea45800 next 169 of size 256
2021-11-16 04:05:07.161125: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cea45900 next 170 of size 65536
2021-11-16 04:05:07.161134: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cea55900 next 171 of size 512
2021-11-16 04:05:07.161144: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cea55b00 next 172 of size 331776
2021-11-16 04:05:07.161155: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceaa6b00 next 173 of size 512
2021-11-16 04:05:07.161163: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceaa6d00 next 174 of size 221184
2021-11-16 04:05:07.161171: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceadcd00 next 175 of size 256
2021-11-16 04:05:07.161179: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceadce00 next 176 of size 65536
2021-11-16 04:05:07.161186: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceaece00 next 177 of size 256
2021-11-16 04:05:07.161195: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceaecf00 next 178 of size 65536
2021-11-16 04:05:07.161203: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceafcf00 next 179 of size 256
2021-11-16 04:05:07.161215: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceafd000 next 180 of size 49152
2021-11-16 04:05:07.161223: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb09000 next 181 of size 256
2021-11-16 04:05:07.161231: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb09100 next 182 of size 147456
2021-11-16 04:05:07.161240: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb2d100 next 183 of size 256
2021-11-16 04:05:07.161250: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb2d200 next 184 of size 49152
2021-11-16 04:05:07.161259: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb39200 next 185 of size 512
2021-11-16 04:05:07.161271: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb39400 next 186 of size 331776
2021-11-16 04:05:07.161281: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb8a400 next 187 of size 512
2021-11-16 04:05:07.161289: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93ceb8a600 next 188 of size 221184
2021-11-16 04:05:07.161297: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cebc0600 next 189 of size 256
2021-11-16 04:05:07.161305: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cebc0700 next 190 of size 49152
2021-11-16 04:05:07.161314: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cebcc700 next 191 of size 256
2021-11-16 04:05:07.161323: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cebcc800 next 192 of size 24576
2021-11-16 04:05:07.161333: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cebd2800 next 193 of size 768
2021-11-16 04:05:07.161343: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cebd2b00 next 194 of size 442368
2021-11-16 04:05:07.161351: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cec3eb00 next 195 of size 256
2021-11-16 04:05:07.161360: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cec3ec00 next 196 of size 16384
2021-11-16 04:05:07.161369: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cec42c00 next 197 of size 256
2021-11-16 04:05:07.161379: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cec42d00 next 198 of size 6144
2021-11-16 04:05:07.161390: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cec44500 next 199 of size 256
2021-11-16 04:05:07.161399: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cec44600 next 200 of size 256
2021-11-16 04:05:07.161409: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cec44700 next 201 of size 20127744
2021-11-16 04:05:07.161418: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76700 next 202 of size 256
2021-11-16 04:05:07.161428: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76800 next 203 of size 256
2021-11-16 04:05:07.161437: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76900 next 204 of size 256
2021-11-16 04:05:07.161447: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76a00 next 205 of size 256
2021-11-16 04:05:07.161456: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76b00 next 206 of size 256
2021-11-16 04:05:07.161465: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76c00 next 207 of size 256
2021-11-16 04:05:07.161473: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76d00 next 208 of size 256
2021-11-16 04:05:07.161480: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76e00 next 209 of size 256
2021-11-16 04:05:07.161488: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff76f00 next 210 of size 256
2021-11-16 04:05:07.161496: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77000 next 211 of size 256
2021-11-16 04:05:07.161504: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77100 next 212 of size 256
2021-11-16 04:05:07.161513: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77200 next 213 of size 256
2021-11-16 04:05:07.161521: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77300 next 214 of size 256
2021-11-16 04:05:07.161529: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77400 next 215 of size 256
2021-11-16 04:05:07.161537: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77500 next 216 of size 256
2021-11-16 04:05:07.161545: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77600 next 217 of size 256
2021-11-16 04:05:07.161553: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77700 next 218 of size 256
2021-11-16 04:05:07.161561: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77800 next 219 of size 256
2021-11-16 04:05:07.161568: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77900 next 220 of size 256
2021-11-16 04:05:07.161576: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77a00 next 221 of size 256
2021-11-16 04:05:07.161584: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77b00 next 222 of size 256
2021-11-16 04:05:07.161593: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77c00 next 223 of size 256
2021-11-16 04:05:07.161601: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77d00 next 224 of size 256
2021-11-16 04:05:07.161610: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77e00 next 225 of size 256
2021-11-16 04:05:07.161618: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff77f00 next 226 of size 256
2021-11-16 04:05:07.161627: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78000 next 227 of size 256
2021-11-16 04:05:07.161636: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78100 next 228 of size 256
2021-11-16 04:05:07.161645: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78200 next 229 of size 256
2021-11-16 04:05:07.161655: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78300 next 230 of size 256
2021-11-16 04:05:07.161664: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78400 next 231 of size 256
2021-11-16 04:05:07.161673: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78500 next 232 of size 256
2021-11-16 04:05:07.161682: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78600 next 233 of size 256
2021-11-16 04:05:07.161690: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78700 next 234 of size 256
2021-11-16 04:05:07.161698: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78800 next 235 of size 256
2021-11-16 04:05:07.161705: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78900 next 236 of size 256
2021-11-16 04:05:07.161713: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78a00 next 237 of size 256
2021-11-16 04:05:07.161721: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78b00 next 238 of size 256
2021-11-16 04:05:07.161729: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78c00 next 239 of size 256
2021-11-16 04:05:07.161737: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78d00 next 240 of size 256
2021-11-16 04:05:07.161746: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78e00 next 241 of size 256
2021-11-16 04:05:07.161754: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff78f00 next 242 of size 256
2021-11-16 04:05:07.161762: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79000 next 243 of size 256
2021-11-16 04:05:07.161770: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79100 next 244 of size 256
2021-11-16 04:05:07.161778: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79200 next 245 of size 256
2021-11-16 04:05:07.161786: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79300 next 246 of size 256
2021-11-16 04:05:07.161794: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79400 next 247 of size 256
2021-11-16 04:05:07.161801: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79500 next 248 of size 256
2021-11-16 04:05:07.161810: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79600 next 249 of size 256
2021-11-16 04:05:07.161822: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79700 next 250 of size 256
2021-11-16 04:05:07.161830: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79800 next 251 of size 256
2021-11-16 04:05:07.161838: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79900 next 252 of size 256
2021-11-16 04:05:07.161847: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79a00 next 253 of size 256
2021-11-16 04:05:07.161854: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79b00 next 254 of size 256
2021-11-16 04:05:07.161862: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79c00 next 255 of size 256
2021-11-16 04:05:07.161871: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79d00 next 256 of size 256
2021-11-16 04:05:07.161880: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79e00 next 257 of size 256
2021-11-16 04:05:07.161888: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff79f00 next 258 of size 256
2021-11-16 04:05:07.161895: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a000 next 259 of size 256
2021-11-16 04:05:07.161902: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a100 next 260 of size 256
2021-11-16 04:05:07.161910: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a200 next 261 of size 256
2021-11-16 04:05:07.161918: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a300 next 262 of size 256
2021-11-16 04:05:07.161926: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a400 next 263 of size 256
2021-11-16 04:05:07.161935: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a500 next 264 of size 256
2021-11-16 04:05:07.161943: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a600 next 265 of size 256
2021-11-16 04:05:07.161950: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a700 next 266 of size 256
2021-11-16 04:05:07.161958: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a800 next 267 of size 256
2021-11-16 04:05:07.161966: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7a900 next 268 of size 256
2021-11-16 04:05:07.161975: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7aa00 next 269 of size 256
2021-11-16 04:05:07.161984: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7ab00 next 270 of size 256
2021-11-16 04:05:07.161991: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7ac00 next 271 of size 256
2021-11-16 04:05:07.161998: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7ad00 next 272 of size 256
2021-11-16 04:05:07.162005: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7ae00 next 273 of size 256
2021-11-16 04:05:07.162011: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7af00 next 274 of size 256
2021-11-16 04:05:07.162018: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b000 next 275 of size 256
2021-11-16 04:05:07.162024: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b100 next 276 of size 256
2021-11-16 04:05:07.162031: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b200 next 277 of size 256
2021-11-16 04:05:07.162039: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b300 next 278 of size 256
2021-11-16 04:05:07.162044: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b400 next 279 of size 256
2021-11-16 04:05:07.162048: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b500 next 280 of size 256
2021-11-16 04:05:07.162052: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b600 next 281 of size 256
2021-11-16 04:05:07.162056: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b700 next 282 of size 256
2021-11-16 04:05:07.162061: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b800 next 283 of size 256
2021-11-16 04:05:07.162065: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7b900 next 284 of size 256
2021-11-16 04:05:07.162069: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7ba00 next 285 of size 256
2021-11-16 04:05:07.162074: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7bb00 next 286 of size 256
2021-11-16 04:05:07.162078: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7bc00 next 287 of size 256
2021-11-16 04:05:07.162083: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7bd00 next 288 of size 256
2021-11-16 04:05:07.162087: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7be00 next 289 of size 256
2021-11-16 04:05:07.162092: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7bf00 next 290 of size 256
2021-11-16 04:05:07.162096: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c000 next 291 of size 256
2021-11-16 04:05:07.162101: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c100 next 292 of size 256
2021-11-16 04:05:07.162105: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c200 next 293 of size 256
2021-11-16 04:05:07.162109: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c300 next 294 of size 256
2021-11-16 04:05:07.162114: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c400 next 295 of size 256
2021-11-16 04:05:07.162118: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c500 next 296 of size 256
2021-11-16 04:05:07.162122: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c600 next 297 of size 256
2021-11-16 04:05:07.162126: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c700 next 298 of size 256
2021-11-16 04:05:07.162131: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c800 next 299 of size 256
2021-11-16 04:05:07.162138: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7c900 next 300 of size 256
2021-11-16 04:05:07.162144: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7ca00 next 301 of size 256
2021-11-16 04:05:07.162151: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7cb00 next 302 of size 256
2021-11-16 04:05:07.162158: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7cc00 next 303 of size 1280
2021-11-16 04:05:07.162164: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f93cff7d100 next 305 of size 1024
2021-11-16 04:05:07.162170: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7d500 next 304 of size 256
2021-11-16 04:05:07.162178: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93cff7d600 next 321 of size 1433600
2021-11-16 04:05:07.162184: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93d00db600 next 314 of size 2457600
2021-11-16 04:05:07.162191: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f93d0333600 next 322 of size 4495616
2021-11-16 04:05:07.162198: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93d077cf00 next 315 of size 1576960
2021-11-16 04:05:07.162202: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93d08fdf00 next 307 of size 69376
2021-11-16 04:05:07.162207: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93d090ee00 next 317 of size 24093184
2021-11-16 04:05:07.162211: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93d2009000 next 308 of size 30720
2021-11-16 04:05:07.162216: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f93d2010800 next 323 of size 17650944
2021-11-16 04:05:07.162221: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f93d30e5d00 next 18446744073709551615 of size 1488377344
2021-11-16 04:05:07.162225: I tensorflow/core/common_runtime/bfc_allocator.cc:914] Summary of in-use Chunks by size:
2021-11-16 04:05:07.162235: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 140 Chunks of size 256 totalling 35.0KiB
2021-11-16 04:05:07.162241: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 32 Chunks of size 512 totalling 16.0KiB
2021-11-16 04:05:07.162247: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 17 Chunks of size 768 totalling 12.8KiB
2021-11-16 04:05:07.162252: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 10 Chunks of size 1024 totalling 10.0KiB
2021-11-16 04:05:07.162257: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 1280 totalling 6.2KiB
2021-11-16 04:05:07.162262: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1536 totalling 3.0KiB
2021-11-16 04:05:07.162267: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2048 totalling 2.0KiB
2021-11-16 04:05:07.162272: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 2304 totalling 11.2KiB
2021-11-16 04:05:07.162277: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 4864 totalling 9.5KiB
2021-11-16 04:05:07.162281: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 6144 totalling 6.0KiB
2021-11-16 04:05:07.162286: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 8704 totalling 8.5KiB
2021-11-16 04:05:07.162291: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 16384 totalling 16.0KiB
2021-11-16 04:05:07.162296: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 24576 totalling 24.0KiB
2021-11-16 04:05:07.162300: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 30720 totalling 30.0KiB
2021-11-16 04:05:07.162305: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 49152 totalling 144.0KiB
2021-11-16 04:05:07.162310: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 65536 totalling 320.0KiB
2021-11-16 04:05:07.162315: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 69376 totalling 67.8KiB
2021-11-16 04:05:07.162320: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 81920 totalling 80.0KiB
2021-11-16 04:05:07.162325: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 110592 totalling 108.0KiB
2021-11-16 04:05:07.162330: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 131072 totalling 128.0KiB
2021-11-16 04:05:07.162334: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 147456 totalling 288.0KiB
2021-11-16 04:05:07.162339: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 163840 totalling 160.0KiB
2021-11-16 04:05:07.162344: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 13 Chunks of size 221184 totalling 2.74MiB
2021-11-16 04:05:07.162349: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 248832 totalling 243.0KiB
2021-11-16 04:05:07.162354: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 262144 totalling 256.0KiB
2021-11-16 04:05:07.162359: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 7 Chunks of size 294912 totalling 1.97MiB
2021-11-16 04:05:07.162364: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 331776 totalling 972.0KiB
2021-11-16 04:05:07.162369: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 368640 totalling 720.0KiB
2021-11-16 04:05:07.162373: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 7 Chunks of size 442368 totalling 2.95MiB
2021-11-16 04:05:07.162378: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 516096 totalling 504.0KiB
2021-11-16 04:05:07.162383: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 524288 totalling 1.00MiB
2021-11-16 04:05:07.162387: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 589824 totalling 1.12MiB
2021-11-16 04:05:07.162392: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 655360 totalling 640.0KiB
2021-11-16 04:05:07.162397: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 737280 totalling 2.11MiB
2021-11-16 04:05:07.162401: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 786432 totalling 2.25MiB
2021-11-16 04:05:07.162406: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 3 Chunks of size 884736 totalling 2.53MiB
2021-11-16 04:05:07.162411: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 921600 totalling 900.0KiB
2021-11-16 04:05:07.162415: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1048576 totalling 1.00MiB
2021-11-16 04:05:07.162420: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1105920 totalling 1.05MiB
2021-11-16 04:05:07.162425: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1179648 totalling 2.25MiB
2021-11-16 04:05:07.162429: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1290240 totalling 1.23MiB
2021-11-16 04:05:07.162434: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1327104 totalling 1.27MiB
2021-11-16 04:05:07.162439: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1433600 totalling 1.37MiB
2021-11-16 04:05:07.162443: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1441792 totalling 2.75MiB
2021-11-16 04:05:07.162448: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1548288 totalling 1.48MiB
2021-11-16 04:05:07.162454: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1576960 totalling 1.50MiB
2021-11-16 04:05:07.162458: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1769472 totalling 1.69MiB
2021-11-16 04:05:07.162463: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1806336 totalling 3.45MiB
2021-11-16 04:05:07.162468: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 2211840 totalling 4.22MiB
2021-11-16 04:05:07.162472: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2359296 totalling 2.25MiB
2021-11-16 04:05:07.162477: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2457600 totalling 2.34MiB
2021-11-16 04:05:07.162482: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2515968 totalling 2.40MiB
2021-11-16 04:05:07.162487: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 4718592 totalling 4.50MiB
2021-11-16 04:05:07.162491: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 5031936 totalling 9.60MiB
2021-11-16 04:05:07.162496: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 5660928 totalling 5.40MiB
2021-11-16 04:05:07.162501: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 10063872 totalling 9.60MiB
2021-11-16 04:05:07.162506: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 17650944 totalling 16.83MiB
2021-11-16 04:05:07.162511: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 20127744 totalling 19.20MiB
2021-11-16 04:05:07.162560: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 24093184 totalling 22.98MiB
2021-11-16 04:05:07.162577: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 140.61MiB
2021-11-16 04:05:07.162586: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 1640313600 memory_limit_: 1640313675 available bytes: 75 curr_region_allocation_bytes_: 3280627712
2021-11-16 04:05:07.162599: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats:
Limit: 1640313675
InUse: 147439616
MaxInUse: 1640312576
NumAllocs: 1200
MaxAllocSize: 1506028288

2021-11-16 04:05:07.162620: W tensorflow/core/common_runtime/bfc_allocator.cc:424] **********__________________________________________________________________________________________

Would like to get some insights on it

Hi @santoshr24 ,
Please try to tune a smaller tf_gpu_memory_fraction in the range of [0.2, 0.6] in the config file to check if it can help this issue

Thanks!

Hey @mchi, i’m running the example as it was, so tf_gpu_memory_fractionwas initially 0.4, I tried it with 0.2 and still doesn’t process further

We don’t have RTX 3050 Ti in hand,
Could you try higher value,e.g. 0.6 or 0.8?

Thanks!

Sure @mchi , with 0.6 and 0.8, it throws this error warning.

I1116 16:12:31.233815 191 metrics.cc:219] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU
I1116 16:12:31.302774 191 pinned_memory_manager.cc:199] Pinned memory pool is created at ‘0x7f0744000000’ with size 268435456
I1116 16:12:31.303075 191 cuda_memory_manager.cc:99] CUDA memory pool is created on device 0 with size 67108864
I1116 16:12:31.303648 191 server.cc:141]
±--------±-------±-----+
| Backend | Config | Path |
±--------±-------±-----+
±--------±-------±-----+

I1116 16:12:31.303664 191 server.cc:184]
±------±--------±-------+
| Model | Version | Status |
±------±--------±-------+
±------±--------±-------+

I1116 16:12:31.303740 191 tritonserver.cc:1620]
±---------------------------------±-------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±-------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.5.0 |
| server_extensions | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memo |
| | ry binary_tensor_data statistics |
| model_repository_path[0] | /opt/nvidia/deepstream/deepstream-5.1/samples/trtis_model_repo |
| model_control_mode | MODE_EXPLICIT |
| strict_model_config | 0 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±-------------------------------------------------------------------------------------------------------------------+

I1116 16:12:31.304749 191 model_repository_manager.cc:810] loading: ssd_inception_v2_coco_2018_01_28:1
2021-11-16 16:12:31.400445: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I1116 16:12:31.437305 191 tensorflow.cc:1876] TRITONBACKEND_Initialize: tensorflow
I1116 16:12:31.437335 191 tensorflow.cc:1889] Triton TRITONBACKEND API version: 1.0
I1116 16:12:31.437341 191 tensorflow.cc:1895] ‘tensorflow’ TRITONBACKEND API version: 1.0
I1116 16:12:31.437345 191 tensorflow.cc:1916] backend configuration:
{“cmdline”:{“allow-soft-placement”:“true”,“gpu-memory-fraction”:“0.600000”}}
I1116 16:12:31.437398 191 tensorflow.cc:1978] TRITONBACKEND_ModelInitialize: ssd_inception_v2_coco_2018_01_28 (version 1)
I1116 16:12:31.438772 191 tensorflow.cc:2028] TRITONBACKEND_ModelInstanceInitialize: ssd_inception_v2_coco_2018_01_28_0 (GPU device 0)
2021-11-16 16:12:31.760330: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3110400000 Hz
2021-11-16 16:12:31.760829: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f06c19ead50 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-11-16 16:12:31.760864: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-11-16 16:12:31.760983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-11-16 16:12:31.761098: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-16 16:12:31.761521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties:
name: NVIDIA GeForce RTX 3050 Ti Laptop GPU major: 8 minor: 6 memoryClockRate(GHz): 1.485
pciBusID: 0000:01:00.0
2021-11-16 16:12:31.761554: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-11-16 16:12:31.761605: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-11-16 16:12:31.761628: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-11-16 16:12:31.761652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-11-16 16:12:31.761678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-11-16 16:12:31.761703: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-11-16 16:12:31.761729: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-11-16 16:12:31.761802: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-16 16:12:31.762216: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-16 16:12:31.762592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0
2021-11-16 16:12:31.762752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-11-16 16:12:31.762773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0
2021-11-16 16:12:31.762785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N
2021-11-16 16:12:31.762873: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-16 16:12:31.763258: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-16 16:12:31.763612: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-16 16:12:31.763958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2346 MB memory) → physical GPU (device: 0, name: NVIDIA GeForce RTX 3050 Ti Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6)
2021-11-16 16:12:31.765778: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f06c1de9630 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-11-16 16:12:31.765827: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3050 Ti Laptop GPU, Compute Capability 8.6
I1116 16:12:31.911459 191 model_repository_manager.cc:983] successfully loaded ‘ssd_inception_v2_coco_2018_01_28’ version 1
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:5 initialized model: ssd_inception_v2_coco_2018_01_28
W1116 16:12:33.234304 191 metrics.cc:283] failed to get power limit for GPU 0: Not Supported
2021-11-16 16:12:34.480281: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
W1116 16:12:35.236046 191 metrics.cc:283] failed to get power limit for GPU 0: Not Supported
2021-11-16 16:12:35.795597: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-11-16 16:12:35.797863: E tensorflow/stream_executor/cuda/cuda_blas.cc:239] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-11-16 16:12:35.798050: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at conv_ops.cc:1006 : Not found: No algorithm worked!
ERROR: infer_trtis_server.cpp:258 Triton: TritonServer response error received., triton_err_str:Internal, err_msg:2 root error(s) found.
(0) Not found: No algorithm worked!
[[{{node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d/depthwise}}]]
[[Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Identity/_64]]
(1) Not found: No algorithm worked!
[[{{node FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d/depthwise}}]]
0 successful operations.
0 derived errors ignored.
ERROR: infer_trtis_backend.cpp:586 TRTIS server failed to parse response with request-id:0 model:
ERROR: infer_trtis_backend.cpp:341 failed to specify dims after running inference failed on model:ssd_inception_v2_coco_2018_01_28, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:07.490120994 191 0x3e50300 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger: nvinferserver[UID 5]: Error in specifyBackendDims() <infer_trtis_context.cpp:174> [UID = 5]: failed to specify input dims trtis backend for model:ssd_inception_v2_coco_2018_01_28, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:07.490155086 191 0x3e50300 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger: nvinferserver[UID 5]: Error in createNNBackend() <infer_trtis_context.cpp:251> [UID = 5]: failed to specify trtis backend input dims for model:ssd_inception_v2_coco_2018_01_28, nvinfer error:NVDSINFER_TRTIS_ERROR
I1116 16:12:35.798709 191 model_repository_manager.cc:837] unloading: ssd_inception_v2_coco_2018_01_28:1
I1116 16:12:35.798957 191 tensorflow.cc:2063] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1116 16:12:35.799558 191 server.cc:280] Waiting for in-flight requests to complete.
I1116 16:12:35.799579 191 server.cc:295] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I1116 16:12:35.849218 191 tensorflow.cc:2001] TRITONBACKEND_ModelFinalize: delete model state
I1116 16:12:35.849912 191 model_repository_manager.cc:966] successfully unloaded ‘ssd_inception_v2_coco_2018_01_28’ version 1
I1116 16:12:36.799703 191 server.cc:295] Timeout 29: Found 0 live models and 0 in-flight non-inference requests
0:00:08.491415384 191 0x3e50300 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger: nvinferserver[UID 5]: Error in initialize() <infer_base_context.cpp:81> [UID = 5]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:08.491461527 191 0x3e50300 WARN nvinferserver gstnvinferserver_impl.cpp:439:start: error: Failed to initialize InferTrtIsContext
0:00:08.491475088 191 0x3e50300 WARN nvinferserver gstnvinferserver_impl.cpp:439:start: error: Config file path: dstest_ssd_nopostprocess.txt
0:00:08.492134498 191 0x3e50300 WARN nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start: error: gstnvinferserver_impl start failed
Error: gst-resource-error-quark: Failed to initialize InferTrtIsContext (1): gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline0/GstNvInferServer:primary-inference:
Config file path: dstest_ssd_nopostprocess.txt
W1116 16:12:37.237786 191 metrics.cc:283] failed to get power limit for GPU 0: Not Supported

Seems RTX 3050 Ti only 4GB device memory, this should be too small to run triton.

You may could take a try DS6.0GA

1 Like

Sure, thank you will try doing that

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.