Modified sample object detector tracker

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
[*] DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
[*] Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
[*] DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
[*] other, 2.0.0

Host Machine Version
[*] native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hi,
I am trying to run the face detector model (blazeface model , used by mediapipe) blaze_face_short_range.zip (248.1 KB)

I am trying to modify the code so that the detections happen. I am following implementation from here to get the preprocessing and postprocessing steps.

For the preprocessing, I used
blaze_face_short_range.trt.json.txt (350 Bytes)

Also, attaching the tensorrt model I obtained after compiling using trtexec (the compilation failed with ./TensorRTOptimizationTool.
blaze_face_short_range.zip (331.5 KB)

The post processing steps, I followed from the github implementation I had shared above.
The code I am using is a modification of sample object detector tracker.
The code is running, and I am printing out the box sizes, and confidence scores.
The confidence scores are coming low, so I am using a lower threshold.
The boxes are also coming at lower sizes.

I tried disabling the feature tracker as well.

Attaching the code here
main.zip (9.8 KB)

Thanks,
Gokul

Dear @gokul.soman,
You need to use the model generated using tensorRT_Optimization tool to integrate your model into DW DNN framework. Could you share the used command and full log.

Hi,

this is the log when using tensorRT_optimization tool

nvidia@tegra-ubuntu:/usr/local/driveworks/tools/dnn$ sudo ./tensorRT_optimization --onnxFile=/home/nvidia/gokul/blaze_face_short_range.onnx --out=~/gokul/optimized/face_detector_trtot.bin --modelType=onnx
[27-02-2024 06:01:51] WARNING: ExplicitBatch is enabled by default for ONNX models.
[27-02-2024 06:01:51] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3
Use model.batch[BATCH_SIZE].onnx (eg. model.batch8.onnx with ONNX batch size 8) with accompanying batch_sizes in configuration.json instead.
[27-02-2024 06:01:52] DNNGenerator: Initializing TensorRT generation on model /home/nvidia/gokul/blaze_face_short_range.onnx.
[27-02-2024 06:01:52] onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[27-02-2024 06:01:52] DNNGenerator: Input “input”: 1x3x128x128
[27-02-2024 06:01:52] DNNGenerator: Output “regressors”: 1x896x16
[27-02-2024 06:01:52] DNNGenerator: Output “classificators”: 1x896x1
[27-02-2024 06:02:15] Releasing Driveworks SDK Context
Error: DW_FILE_INVALID: Cannot open file for writing. Please ensure that you have write permissions.

Dear @gokul.soman,
Looks like some issue with file permission. Could you check running from home folder with out sudo. Also, make sure the output folder is empty and has write permission.

Also, any reason for using DRIVE OS 6.0.6 and not latest release?

Dear @gokul.soman,
I quickly tested the model on DRIVE OS 6.0.8.1 release. I don’t see any errors as such

nvidia@tegra-ubuntu:~/siva$ /usr/local/driveworks/tools/dnn/tensorRT_optimization --onnxFile=/home/nvidia/siva/blaze_face_short_range.onnx --out=/home/nvidia/siva/blaze.bin
ProgramArguments: Missing argument 'outputBlobs' requested
ProgramArguments: Missing argument 'inputBlobs' requested
ProgramArguments: Missing argument 'inputDims' requested
[27-02-2024 07:28:13] WARNING: ExplicitBatch is enabled by default for ONNX models.
[27-02-2024 07:28:13] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3
                      Use model.batch[BATCH_SIZE].onnx (eg. `model.batch8.onnx` with ONNX batch size 8) with accompanying batch_sizes in configuration.json instead.
[27-02-2024 07:28:15] DNNGenerator: Initializing TensorRT generation on model /home/nvidia/siva/blaze_face_short_range.onnx.
[27-02-2024 07:28:16] onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[27-02-2024 07:28:16] DNNGenerator: Input "input": 1x3x128x128
[27-02-2024 07:28:16] DNNGenerator: Output "regressors": 1x896x16
[27-02-2024 07:28:16] DNNGenerator: Output "classificators": 1x896x1
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 0: 2.965376 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 1: 0.391840 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 2: 0.392512 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 3: 0.395680 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 4: 0.423296 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 5: 0.392576 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 6: 0.391072 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 7: 0.390976 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 8: 0.387328 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 9: 0.393952 ms.
[27-02-2024 07:28:43] DNNValidator: Average over 10 runs is 0.652461 ms.
[27-02-2024 07:28:43] [27-02-2024 07:28:43] Releasing Driveworks SDK Context

Hi @SivaRamaKrishnaNV ,
I moved the binary into a folder in home directory and ran the same. but then again getting different error

nvidia@tegra-ubuntu:~/gokul$ ./tensorRT_optimization --modelType=onnx --onnxFile=blaze_face_short_range.onnx --out=test_folder/test.bin
[27-02-2024 07:33:52] WARNING: ExplicitBatch is enabled by default for ONNX models.
[27-02-2024 07:33:52] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3
Use model.batch[BATCH_SIZE].onnx (eg. model.batch8.onnx with ONNX batch size 8) with accompanying batch_sizes in configuration.json instead.
[27-02-2024 07:34:05] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See CUDA_MODULE_LOADING in CUDA C++ Programming Guide
[27-02-2024 07:34:05] DNNGenerator: Initializing TensorRT generation on model blaze_face_short_range.onnx.
[27-02-2024 07:34:05] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[27-02-2024 07:34:05] DNNGenerator: Input “input”: 1x3x128x128
[27-02-2024 07:34:05] DNNGenerator: Output “regressors”: 1x896x16
[27-02-2024 07:34:05] DNNGenerator: Output “classificators”: 1x896x1
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:06] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:07] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:08] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:09] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:10] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:11] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:12] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:13] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:14] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:15] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:16] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] Caught cuda error during timing, format rejected.
[27-02-2024 07:34:17] GPU error during getBestTactic: conv2d + activation : no kernel image is available for execution on the device
[27-02-2024 07:34:17] 1: [convBaseRunner.cpp::execute::271] Error Code 1: Cask (Cask convolution execution)
[27-02-2024 07:34:17] 1: [checkMacros.cpp::catchCudaError::202] Error Code 1: Cuda Runtime (no kernel image is available for execution on the device)
[27-02-2024 07:34:18] 2: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[27-02-2024 07:34:18] Releasing Driveworks SDK Context
Error: DW_INTERNAL_ERROR: DNNGenerator: Network build and serialization failed.

So, is the solution to flash newer version of driveOS?

Thanks,
Gokul

I copied the binary to new folder and tested. I don’t see any issue with model generation.

nvidia@tegra-ubuntu:~/siva$ cp /usr/local/driveworks/tools/dnn/tensorRT_optimization .
nvidia@tegra-ubuntu:~/siva$ ./tensorRT_optimization --onnxFile= blaze
blaze.bin                    blaze_face_short_range.onnx
nvidia@tegra-ubuntu:~/siva$ ./tensorRT_optimization --onnxFile=blaze_face_short_range.onnx --out=blaze-new.bin
ProgramArguments: Missing argument 'outputBlobs' requested
ProgramArguments: Missing argument 'inputBlobs' requested
ProgramArguments: Missing argument 'inputDims' requested
[27-02-2024 07:44:27] WARNING: ExplicitBatch is enabled by default for ONNX models.
[27-02-2024 07:44:27] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3
                      Use model.batch[BATCH_SIZE].onnx (eg. `model.batch8.onnx` with ONNX batch size 8) with accompanying batch_sizes in configuration.json instead.
[27-02-2024 07:44:30] DNNGenerator: Initializing TensorRT generation on model blaze_face_short_range.onnx.
[27-02-2024 07:44:30] onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[27-02-2024 07:44:30] DNNGenerator: Input "input": 1x3x128x128
[27-02-2024 07:44:30] DNNGenerator: Output "regressors": 1x896x16
[27-02-2024 07:44:30] DNNGenerator: Output "classificators": 1x896x1
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 0: 2.996320 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 1: 0.413728 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 2: 0.400288 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 3: 0.440544 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 4: 0.386688 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 5: 0.388768 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 6: 0.395232 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 7: 0.385344 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 8: 0.383104 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 9: 0.382976 ms.
[27-02-2024 07:44:56] DNNValidator: Average over 10 runs is 0.657299 ms.
[27-02-2024 07:44:56] [27-02-2024 07:44:56] Releasing Driveworks SDK Context

I am not sure if the issue related to your environment. We recommend to upgrade to new release if you don’t have any dependency on DRIVE OS version.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.