Description
“[jitConvRunner.cpp::onShapeChange::285] Error Code 1: Cask (no further information)” error happens when I call the enqueue function in the separate thread.
If I put cudaStreamSynchronize() before calling enqueue function, the error does not occurs.
Why does this issue occur?
Environment
TensorRT Version: 8.6.2 (Jetpack 6.0)
GPU Type: NVIDIA Jetson AGX Orin 32GB
Nvidia Driver Version: not available
CUDA Version: 12.2
CUDNN Version: 8.9.4
Operating System + Version: Ubuntu 22.04
Relevant Files
enqueueError.tar.gz (161.2 KB)
Engine file link
Steps To Reproduce
- Download enqueueError.tar.gz (161.2 KB) and Engine file link.
- Extract both files and place yolov3-static.rt to the enqueueError directory.
- Compile the sample code with the following commands
mkdir build ; cd build ; cmake .. ; make -j ; cd .. - run the following command
./build/example1 - The printed result shows the following:
engine name: yolov3-static.rt
[:main:118]
TENSORRT LOG: Loaded engine size: 119 MiB
TENSORRT LOG: Deserialization required 35285 microseconds.
TENSORRT LOG: [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +118, now: CPU 0, GPU 118 (MiB)
125791196
[:main:123]
binding_num: 4, profile_num: 1
tensor name: 000_net, isInput: 1
dims[0]: 1, dims[1]: 3, dims[2]: 608, dims[3]: 608
tensor name: 082_convolutional, isInput: 0
dims[0]: 1, dims[1]: 255, dims[2]: 19, dims[3]: 19
tensor name: 094_convolutional, isInput: 0
dims[0]: 1, dims[1]: 255, dims[2]: 38, dims[3]: 38
tensor name: 106_convolutional, isInput: 0
dims[0]: 1, dims[1]: 255, dims[2]: 76, dims[3]: 76
[:main:138]
TENSORRT LOG: Total per-runner device persistent memory is 1024
TENSORRT LOG: Total per-runner host persistent memory is 450832
TENSORRT LOG: Allocated activation device memory of size 55079936
TENSORRT LOG: [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +52, now: CPU 0, GPU 170 (MiB)
TENSORRT LOG: CUDA lazy loading is enabled.
dims[0]: 1, dims[1]: 3, dims[2]: 608, dims[3]: 608
dims[0]: 1, dims[1]: 255, dims[2]: 19, dims[3]: 19
dims[0]: 1, dims[1]: 255, dims[2]: 38, dims[3]: 38
dims[0]: 1, dims[1]: 255, dims[2]: 76, dims[3]: 76
[:main:162]
[:main:171] name: 000_net, isOnDevice: 1
[:main:171] name: 082_convolutional, isOnDevice: 1
[:main:171] name: 094_convolutional, isOnDevice: 1
[:main:171] name: 106_convolutional, isOnDevice: 1
[:main:189]
[:runInstanceThread:101]
TENSORRT LOG: 1: [jitConvRunner.cpp::onShapeChange::285] Error Code 1: Cask (no further information)
example1: /sdcard/jrkim/np/program/enqueueError/example1.cc:103: void runInstanceThread(void*): Assertion `result' failed.
Aborted (core dumped)
- You can find the following error message in the printed result
[jitConvRunner.cpp::onShapeChange::285] Error Code 1: Cask (no further information)
Note that This error does not occur if you uncomment the following line shown in line 99 of example1.cc.
//cudaStreamSynchronize(stream);