Description
I am trying to convert my resnet100 for classification tasks into the INT8 format. I have the inference code, and validation code, and was able to confirm FP32 and FP16 modes both gave accurate results. However, INT8 gave me a lot of trouble.
After carefully following the image_batcher.py and build_engine.py in TensorRT/samples/python/efficientnet at master · NVIDIA/TensorRT · GitHub, I was able to use identical preprocessing steps (to inference) in the batcher, and I validated the range of the preprocessed input tensors. They seemed correct. The program generated a calibration file with my directory of 5000 images (sample below used less). And it eventually saved an engine to disk. However, when I tried to examine the feature vectors, different input images will produce identical output, and they are incorrect, compared with same code with FP32 and FP16 modes. My evaluation just ended up having 0 accuracy.
My onnx model had batch axis as dynamic, so I added a dynamic profile with batch size ranging from 1 to 256. It didn’t matter what batch size calibration used.
Environment
TensorRT Version : 8.0.3
GPU Type : Tesla T4
Nvidia Driver Version : 450.119.03
CUDA Version : In container cuda-11.5
CUDNN Version :
Operating System + Version : ec2 instance
Python Version (if applicable) : 3.8.10
TensorFlow Version (if applicable) :
PyTorch Version (if applicable) :
Baremetal or Container (if container which image + tag) : Container nvcr.io/nvidia/tensorrt:21.11-py3
Steps To Reproduce
Here is my customization to TensorRT/build_engine.py at master · NVIDIA/TensorRT · GitHub
log.info("Network Description")
for input in inputs:
self.batch_size = args.calib_batch_size
log.info("Input '{}' with shape {} and dtype {}".format(input.name, input.shape, input.dtype))
profile = self.builder.create_optimization_profile()
profile.set_shape(input.name, (1, 3, input.shape[2], input.shape[3]),
(128, 3, input.shape[2], input.shape[3]),
(256, 3, input.shape[2], input.shape[3]))
self.config.add_optimization_profile(profile)
for output in outputs:
log.info("Output '{}' with shape {} and dtype {}".format(output.name, output.shape, output.dtype))
assert self.batch_size > 0
self.builder.max_batch_size = 256
Here is the output from that build script:
[TensorRT] INFO: [MemUsageChange] Init CUDA: CPU +321, GPU +0, now: CPU 342, GPU 252 (MiB)
[TensorRT] WARNING: onnx2trt_utils.cpp:362: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
INFO:EngineBuilder:Network Description
INFO:EngineBuilder:Input 'input.1' with shape (-1, 3, 224, 224) and dtype DataType.FLOAT
INFO:EngineBuilder:Output '1333' with shape (-1, 512) and dtype DataType.FLOAT
INFO:EngineBuilder:Building int8 Engine in /training/models/resnet100.trt
/training/docker-tensorrt-workbench/build_engine.py:209: DeprecationWarning: Use build_serialized_network instead.
with self.builder.build_engine(self.network, self.config) as engine, open(engine_path, "wb") as f:
[TensorRT] INFO: [MemUsageSnapshot] Builder begin: CPU 592 MiB, GPU 254 MiB
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +507, GPU +220, now: CPU 1101, GPU 474 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +114, GPU +52, now: CPU 1215, GPU 526 (MiB)
[TensorRT] WARNING: Calibration Profile is not defined. Running calibration with Profile 0
[TensorRT] INFO: Detected 1 inputs and 1 output network tensors.
[TensorRT] INFO: Total Host Persistent Memory: 22528
[TensorRT] INFO: Total Device Persistent Memory: 0
[TensorRT] INFO: Total Scratch Memory: 0
[TensorRT] INFO: [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 4 MiB
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1738, GPU 1002 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 1739, GPU 1010 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1738, GPU 994 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1738, GPU 978 (MiB)
[TensorRT] INFO: [MemUsageSnapshot] ExecutionContext creation begin: CPU 1738 MiB, GPU 978 MiB
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1738, GPU 986 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1738, GPU 994 (MiB)
[TensorRT] INFO: [MemUsageSnapshot] ExecutionContext creation end: CPU 1738 MiB, GPU 1902 MiB
[TensorRT] INFO: Starting Calibration.
INFO:EngineBuilder:Calibrating image 4 / 40
[TensorRT] INFO: Calibrated batch 0 in 3.5102 seconds.
INFO:EngineBuilder:Calibrating image 8 / 40
[TensorRT] INFO: Calibrated batch 1 in 3.51987 seconds.
INFO:EngineBuilder:Calibrating image 12 / 40
[TensorRT] INFO: Calibrated batch 2 in 3.54024 seconds.
INFO:EngineBuilder:Calibrating image 16 / 40
[TensorRT] INFO: Calibrated batch 3 in 3.54137 seconds.
INFO:EngineBuilder:Calibrating image 20 / 40
[TensorRT] INFO: Calibrated batch 4 in 3.5381 seconds.
INFO:EngineBuilder:Calibrating image 24 / 40
[TensorRT] INFO: Calibrated batch 5 in 3.53881 seconds.
INFO:EngineBuilder:Calibrating image 28 / 40
[TensorRT] INFO: Calibrated batch 6 in 3.56045 seconds.
INFO:EngineBuilder:Calibrating image 32 / 40
[TensorRT] INFO: Calibrated batch 7 in 3.56059 seconds.
INFO:EngineBuilder:Calibrating image 36 / 40
[TensorRT] INFO: Calibrated batch 8 in 3.55971 seconds.
INFO:EngineBuilder:Calibrating image 40 / 40
[TensorRT] INFO: Calibrated batch 9 in 3.56199 seconds.
INFO:EngineBuilder:Finished calibration batches
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1746, GPU 1886 (MiB)
[TensorRT] INFO: Post Processing Calibration data in 38.0515 seconds.
[TensorRT] INFO: Calibration completed in 78.6834 seconds.
[TensorRT] INFO: Writing Calibration Cache for calibrator: TRT-8003-EntropyCalibration2
INFO:EngineBuilder:Writing calibration cache data to: calibration.cache
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1781, GPU 734 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1781, GPU 742 (MiB)
[TensorRT] WARNING: Detected invalid timing cache, setup a local cache instead
[TensorRT] INFO: Detected 1 inputs and 1 output network tensors.
[TensorRT] INFO: Total Host Persistent Memory: 243872
[TensorRT] INFO: Total Device Persistent Memory: 53171200
[TensorRT] INFO: Total Scratch Memory: 512
[TensorRT] INFO: [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 148 MiB, GPU 910 MiB
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1870, GPU 828 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1870, GPU 836 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1870, GPU 820 (MiB)
[TensorRT] INFO: [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1869, GPU 804 (MiB)
Here is my generated calibration cache:
TRT-8003-EntropyCalibration2
input.1: 7f800000
1334: 7f800000
(Unnamed Layer* 1) [Constant]_output: 3bd7b8b7
(Unnamed Layer* 2) [Shuffle]_output: 3bd7b8b7
929: 7f800000
930: 7f800000
1337: 7f800000
(Unnamed Layer* 6) [Constant]_output: 3b0efec6
(Unnamed Layer* 7) [Shuffle]_output: 3b0efec6
934: 7f800000
1340: 7f800000
1343: 7ab458b8
939: 7f800000
940: 7f800000
1346: 7f800000
(Unnamed Layer* 14) [Constant]_output: 3ad4d621
(Unnamed Layer* 15) [Shuffle]_output: 3ad4d621
944: 7f800000
1349: 7a90e807
947: 7f800000
948: 7f800000
1352: 7f800000
(Unnamed Layer* 21) [Constant]_output: 3a969a81
(Unnamed Layer* 22) [Shuffle]_output: 3a969a81
952: 7f800000
1355: 7b035e84
955: 7f800000
956: 7f800000
1358: 7f800000
(Unnamed Layer* 28) [Constant]_output: 3b0ee760
(Unnamed Layer* 29) [Shuffle]_output: 3b0ee760
960: 7f800000
1361: 7a4e9ef7
1364: 7a0ed897
965: 7a805efb
966: 79ff7b20
1367: 7a0bd9a8
(Unnamed Layer* 36) [Constant]_output: 3acaee93
(Unnamed Layer* 37) [Shuffle]_output: 3acaee93
970: 79a74866
1370: 7988f33e
973: 7a24191d
974: 79b4ce9c
1373: 7984a58b
(Unnamed Layer* 43) [Constant]_output: 3aab3324
(Unnamed Layer* 44) [Shuffle]_output: 3aab3324
978: 78901743
1376: 760b50ec
981: 760b50ec
982: 761d6dcb
1379: 75e783b4
(Unnamed Layer* 50) [Constant]_output: 3a8b7a92
(Unnamed Layer* 51) [Shuffle]_output: 3a8b7a92
986: 750782a1
1382: 3ad03e59
989: 3bbd7be6
990: 3aff2390
1385: 3b75658c
(Unnamed Layer* 57) [Constant]_output: 3a9a2797
(Unnamed Layer* 58) [Shuffle]_output: 3a9a2797
994: 3b367975
1388: 3b578819
997: 3c0d3eb2
998: 3b7a7d0e
1391: 3b92bad5
(Unnamed Layer* 64) [Constant]_output: 3a9f9490
(Unnamed Layer* 65) [Shuffle]_output: 3a9f9490
1002: 3b2019b6
1394: 3b81d8d4
1005: 3c50a850
1006: 3ba35a12
1397: 3bc35d71
(Unnamed Layer* 71) [Constant]_output: 3af3ad53
(Unnamed Layer* 72) [Shuffle]_output: 3af3ad53
1010: 3b2e08b5
1400: 3b8c3ecb
1013: 3c5b1b0f
1014: 3b954245
1403: 3ba2990d
(Unnamed Layer* 78) [Constant]_output: 3aca1921
(Unnamed Layer* 79) [Shuffle]_output: 3aca1921
1018: 3b1c5c5f
1406: 3bcbac74
1021: 3c5ee144
1022: 3b990fbf
1409: 3baa9825
(Unnamed Layer* 85) [Constant]_output: 3b05ee7a
(Unnamed Layer* 86) [Shuffle]_output: 3b05ee7a
1026: 3b207e7b
1412: 3b8a2d2d
1029: 3c6ec711
1030: 3ba839f0
1415: 3b90c2e7
(Unnamed Layer* 92) [Constant]_output: 3abfafff
(Unnamed Layer* 93) [Shuffle]_output: 3abfafff
1034: 3b12b5ec
1418: 3b84d275
1037: 3c828356
1038: 3b62060b
1421: 3b82c46e
(Unnamed Layer* 99) [Constant]_output: 3ad0594a
(Unnamed Layer* 100) [Shuffle]_output: 3ad0594a
1042: 3b23cf3f
1424: 3bc19cb1
1045: 3c9c3911
1046: 3b89f6de
1427: 3b8b7f01
(Unnamed Layer* 106) [Constant]_output: 3aa2b279
(Unnamed Layer* 107) [Shuffle]_output: 3aa2b279
1050: 3b0d6831
1430: 3bbff987
1053: 3cb5e90b
1054: 3b989c5e
1433: 3b7f3220
(Unnamed Layer* 113) [Constant]_output: 3aacd63c
(Unnamed Layer* 114) [Shuffle]_output: 3aacd63c
1058: 3b186342
1436: 3ba70bdc
1061: 3cde4ad0
1062: 3bd0bc14
1439: 3bf217bf
(Unnamed Layer* 120) [Constant]_output: 3ad4cfe5
(Unnamed Layer* 121) [Shuffle]_output: 3ad4cfe5
1066: 3bf217bf
1442: 3b567412
1445: 3b1ea081
1071: 3b9bc123
1072: 3ab5bc2e
1448: 3b6c1ffb
(Unnamed Layer* 128) [Constant]_output: 3a8a53d7
(Unnamed Layer* 129) [Shuffle]_output: 3a8a53d7
1076: 3b0f86e4
1451: 3ae20f9c
1079: 3b9b02ba
1080: 3b61be4c
1454: 3b88f7e0
(Unnamed Layer* 135) [Constant]_output: 3a7a4331
(Unnamed Layer* 136) [Shuffle]_output: 3a7a4331
1084: 3ab1e180
1457: 3ac3fe16
1087: 3bbc7fbb
1088: 3b401555
1460: 3b5af4f0
(Unnamed Layer* 142) [Constant]_output: 3a8e7af2
(Unnamed Layer* 143) [Shuffle]_output: 3a8e7af2
1092: 3af870e7
1463: 3ac71e86
1095: 3bcb401c
1096: 3b603f99
1466: 3bd4708f
(Unnamed Layer* 149) [Constant]_output: 3a84c7f1
(Unnamed Layer* 150) [Shuffle]_output: 3a84c7f1
1100: 3bdb0064
1469: 3b16dfcb
1103: 3be5579d
1104: 3b74ecb7
1472: 3ba9e62c
(Unnamed Layer* 156) [Constant]_output: 3a6ebd68
(Unnamed Layer* 157) [Shuffle]_output: 3a6ebd68
1108: 3baefd5f
1475: 3b1ca833
1111: 3be97e83
1112: 3b8e5559
1478: 3bb9d60f
(Unnamed Layer* 163) [Constant]_output: 3a8852d7
(Unnamed Layer* 164) [Shuffle]_output: 3a8852d7
1116: 3b895b6f
1481: 3b35a312
1119: 3bd62fe7
1120: 3b84fbed
1484: 3baef164
(Unnamed Layer* 170) [Constant]_output: 3afc2d20
(Unnamed Layer* 171) [Shuffle]_output: 3afc2d20
1124: 3baef164
1487: 3b81bb3c
1127: 3c0d8b7d
1128: 3b8e2fbd
1490: 3ba869e3
(Unnamed Layer* 177) [Constant]_output: 3a82e480
(Unnamed Layer* 178) [Shuffle]_output: 3a82e480
1132: 3ae0018f
1493: 3b33bcab
1135: 3be3b76a
1136: 3b8dc703
1496: 3b96fe57
(Unnamed Layer* 184) [Constant]_output: 3a8c7e8e
(Unnamed Layer* 185) [Shuffle]_output: 3a8c7e8e
1140: 3b1ec9e1
1499: 3b515594
1143: 3c02dfd2
1144: 3b92e7f4
1502: 3b941d08
(Unnamed Layer* 191) [Constant]_output: 3a8f4047
(Unnamed Layer* 192) [Shuffle]_output: 3a8f4047
1148: 3b1e490d
1505: 3b08736e
1151: 3c194e1c
1152: 3b65e9cf
1508: 3ba6edda
(Unnamed Layer* 198) [Constant]_output: 3a94b4c9
(Unnamed Layer* 199) [Shuffle]_output: 3a94b4c9
1156: 3af0949b
1511: 3aec2aab
1159: 3c19bd98
1160: 3b5d276d
1514: 3b61ef4c
(Unnamed Layer* 205) [Constant]_output: 3a90d257
(Unnamed Layer* 206) [Shuffle]_output: 3a90d257
1164: 3ac0786d
1517: 3aff5b5b
1167: 3bf868e0
1168: 3b262f60
1520: 3b8cca4b
(Unnamed Layer* 212) [Constant]_output: 3a83b99b
(Unnamed Layer* 213) [Shuffle]_output: 3a83b99b
1172: 3ab3616a
1523: 3ac8cd82
1175: 3c299b8e
1176: 3b14345f
1526: 3b832ddd
(Unnamed Layer* 219) [Constant]_output: 3a8d6282
(Unnamed Layer* 220) [Shuffle]_output: 3a8d6282
1180: 3aaa8079
1529: 3ac64a56
1183: 3c1b556f
1184: 3b08705c
1532: 3b434171
(Unnamed Layer* 226) [Constant]_output: 3aa7d55d
(Unnamed Layer* 227) [Shuffle]_output: 3aa7d55d
1188: 3ab8e3b8
1535: 3b0429e5
1191: 3c06cd05
1192: 3b4151d5
1538: 3b810394
(Unnamed Layer* 233) [Constant]_output: 3a9977d9
(Unnamed Layer* 234) [Shuffle]_output: 3a9977d9
1196: 3aa41975
1541: 3acf2a74
1199: 3c25753f
1200: 3b2ee3d3
1544: 3b740030
(Unnamed Layer* 240) [Constant]_output: 3a800fad
(Unnamed Layer* 241) [Shuffle]_output: 3a800fad
1204: 3ad16109
1547: 3b1a54fe
1207: 3c24e6fc
1208: 3b31d579
1550: 3b6d1c5f
(Unnamed Layer* 247) [Constant]_output: 3a810c04
(Unnamed Layer* 248) [Shuffle]_output: 3a810c04
1212: 3a98343e
1553: 3b006418
1215: 3c2aba7e
1216: 3b258b3c
1556: 3b61e917
(Unnamed Layer* 254) [Constant]_output: 3a833126
(Unnamed Layer* 255) [Shuffle]_output: 3a833126
1220: 3ab9ad44
1559: 3b346df2
1223: 3c0c8b8f
1224: 3b279bc7
1562: 3b7c5105
(Unnamed Layer* 261) [Constant]_output: 3a859747
(Unnamed Layer* 262) [Shuffle]_output: 3a859747
1228: 3ac08109
1565: 3b2a6c65
1231: 3c4d5550
1232: 3b2f53ac
1568: 3b49f838
(Unnamed Layer* 268) [Constant]_output: 3a96c0f3
(Unnamed Layer* 269) [Shuffle]_output: 3a96c0f3
1236: 3aa7961d
1571: 3b2b98bd
1239: 3c3e7f44
1240: 3b324498
1574: 3b715eae
(Unnamed Layer* 275) [Constant]_output: 3a899b80
(Unnamed Layer* 276) [Shuffle]_output: 3a899b80
1244: 3abc6eae
1577: 3b50fa3f
1247: 3c4f6214
1248: 3b3df409
1580: 3b6e2524
(Unnamed Layer* 282) [Constant]_output: 3a9bc15c
(Unnamed Layer* 283) [Shuffle]_output: 3a9bc15c
1252: 3a9968a7
1583: 3b45157d
1255: 3c51d5c1
1256: 3b33bfa2
1586: 3b7c72dc
(Unnamed Layer* 289) [Constant]_output: 3a8b47e3
(Unnamed Layer* 290) [Shuffle]_output: 3a8b47e3
1260: 3ac21a64
1589: 3b461591
1263: 3c557fe0
1264: 3b255ca6
1592: 3b72b93c
(Unnamed Layer* 296) [Constant]_output: 3aaa84ce
(Unnamed Layer* 297) [Shuffle]_output: 3aaa84ce
1268: 3a949169
1595: 3b999bf0
1271: 3c5c96a2
1272: 3b2b422f
1598: 3b92f3aa
(Unnamed Layer* 303) [Constant]_output: 3a92430a
(Unnamed Layer* 304) [Shuffle]_output: 3a92430a
1276: 3a8f6092
1601: 3b8d44dc
1279: 3c729db1
1280: 3b42c812
1604: 3ba398b1
(Unnamed Layer* 310) [Constant]_output: 3a9426dd
(Unnamed Layer* 311) [Shuffle]_output: 3a9426dd
1284: 3acf4f80
1607: 3b566ea9
1287: 3c5d79a7
1288: 3b2e0069
1610: 3b8acd90
(Unnamed Layer* 317) [Constant]_output: 3a938d19
(Unnamed Layer* 318) [Shuffle]_output: 3a938d19
1292: 3ade2fe6
1613: 3bc06970
1295: 3c673039
1296: 3b29fe0a
1616: 3b71492c
(Unnamed Layer* 324) [Constant]_output: 3a774f86
(Unnamed Layer* 325) [Shuffle]_output: 3a774f86
1300: 3ae6a4b3
1619: 3b9b0e4b
1303: 3c7e875d
1304: 3b8354c6
1622: 3b6ce6b1
(Unnamed Layer* 331) [Constant]_output: 3ae74fa8
(Unnamed Layer* 332) [Shuffle]_output: 3ae74fa8
1308: 3b1469cb
1625: 3ad35bc4
1628: 3a41e752
1313: 3b195750
1314: 3afa5103
1631: 3bafb5d5
(Unnamed Layer* 339) [Constant]_output: 3ace4739
(Unnamed Layer* 340) [Shuffle]_output: 3ace4739
1318: 3ac85277
1634: 3ae1731b
1321: 3b3926ee
1322: 3b39e7e9
1637: 3b935125
(Unnamed Layer* 346) [Constant]_output: 3b0ad1c8
(Unnamed Layer* 347) [Shuffle]_output: 3b0ad1c8
1326: 3ad43734
1640: 3b0e8937
1329: 3b551758
1330: 3b54c765
1331: 3b54c765
(Unnamed Layer* 363) [Shuffle]_output: 3b54c765
(Unnamed Layer* 364) [Fully Connected]_output: 3b14cb0e
1332: 3b14cb0e
(Unnamed Layer* 374) [Shuffle]_output: 3b14cb0e
(Unnamed Layer* 375) [Scale]_output: 3cb503e3
1333: 3cb503e3
I don’t know how to decipher this.