Hi all,
I am newbie to the NVIDIA AI world. I want to check the mlperf inference resnet50 benchmark result on my GPU A30.
Follow below user guide to setup SUT for running resnet50 benchmark.
However I get RuntimeError(“Building engines failed!”) and I have no idea how to fix it. can anyone give me some suggestion. Thanks
Below is the output.
(mlperf) test@mlperf-inference-test-x86_64:/work$ make run RUN_ARGS=“–benchmarks=resnet50 --scenarios=offline”
make[1]: Entering directory ‘/work’
[2023-05-12 05:35:37,521 main_v2.py:221 INFO] Detected system ID: KnownSystem.AMD_K905
[2023-05-12 05:35:39,726 generate_engines.py:172 INFO] Building engines for resnet50 benchmark in Offline scenario…
[2023-05-12 05:35:39,752 ResNet50.py:36 INFO] Using workspace size: 0
[05/12/2023-05:35:40] [TRT] [I] [MemUsageChange] Init CUDA: CPU +320, GPU +0, now: CPU 354, GPU 642 (MiB)
[05/12/2023-05:35:41] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +414, GPU +132, now: CPU 787, GPU 774 (MiB)
Process Process-1:
Traceback (most recent call last):
- File “/usr/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap*
- self.run()*
- File “/usr/lib/python3.8/multiprocessing/process.py”, line 108, in run*
- self._target(*self._args, *self._kwargs)
- File “/work/code/actionhandler/base.py”, line 185, in subprocess_target*
- return self.action_handler.handle()*
- File “/work/code/actionhandler/generate_engines.py”, line 175, in handle*
- total_engine_build_time += self.build_engine(job)*
- File “/work/code/actionhandler/generate_engines.py”, line 166, in build_engine*
- builder.build_engines()*
- File “/work/code/common/builder.py”, line 170, in build_engines*
- self.initialize()*
- File “/work/code/resnet50/tensorrt/ResNet50.py”, line 78, in initialize*
- rn50_gs = RN50GraphSurgeon(self.model_path,*
- File “/work/code/resnet50/tensorrt/rn50_graphsurgeon.py”, line 328, in init*
- if os.path.exists(self.cache_file):*
- File “/usr/lib/python3.8/genericpath.py”, line 19, in exists*
- os.stat(path)*
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
[2023-05-12 05:35:43,882 generate_engines.py:172 INFO] Building engines for resnet50 benchmark in Offline scenario…
[2023-05-12 05:35:43,899 ResNet50.py:36 INFO] Using workspace size: 0
[05/12/2023-05:35:44] [TRT] [I] [MemUsageChange] Init CUDA: CPU +320, GPU +0, now: CPU 354, GPU 642 (MiB)
[05/12/2023-05:35:45] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +414, GPU +132, now: CPU 787, GPU 774 (MiB)
Process Process-2:
Traceback (most recent call last): - File “/usr/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap*
- self.run()*
- File “/usr/lib/python3.8/multiprocessing/process.py”, line 108, in run*
- self._target(*self._args, *self._kwargs)
- File “/work/code/actionhandler/base.py”, line 185, in subprocess_target*
- return self.action_handler.handle()*
- File “/work/code/actionhandler/generate_engines.py”, line 175, in handle*
- total_engine_build_time += self.build_engine(job)*
- File “/work/code/actionhandler/generate_engines.py”, line 166, in build_engine*
- builder.build_engines()*
- File “/work/code/common/builder.py”, line 170, in build_engines*
- self.initialize()*
- File “/work/code/resnet50/tensorrt/ResNet50.py”, line 78, in initialize*
- rn50_gs = RN50GraphSurgeon(self.model_path,*
- File “/work/code/resnet50/tensorrt/rn50_graphsurgeon.py”, line 328, in init*
- if os.path.exists(self.cache_file):*
- File “/usr/lib/python3.8/genericpath.py”, line 19, in exists*
- os.stat(path)*
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last): - File “/usr/lib/python3.8/runpy.py”, line 194, in _run_module_as_main*
- return _run_code(code, main_globals, None,*
- File “/usr/lib/python3.8/runpy.py”, line 87, in _run_code*
- exec(code, run_globals)*
- File “/work/code/main_v2.py”, line 223, in *
- main(main_args, DETECTED_SYSTEM)*
- File “/work/code/main_v2.py”, line 147, in main*
- dispatch_action(main_args, config_dict, workload_setting)*
- File “/work/code/main_v2.py”, line 194, in dispatch_action*
- handler.run()*
- File “/work/code/actionhandler/base.py”, line 79, in run*
- self.handle_failure()*
- File “/work/code/actionhandler/base.py”, line 182, in handle_failure*
- self.action_handler.handle_failure()*
- File “/work/code/actionhandler/generate_engines.py”, line 183, in handle_failure*
- raise RuntimeError(“Building engines failed!”)*
RuntimeError: Building engines failed!
make[1]: *** [Makefile:694: generate_engines] Error 1
make[1]: Leaving directory ‘/work’
My SUT config
OS :ubuntu 22.04.2
GPU :A30 x2
Driver :510.39.01
Nvidia-ctk :1.13.1
nvidia-smi.txt (14.8 KB)