# Modified sample object detector tracker

**URL:** <https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056>\
**Category:** DRIVE AGX Orin General\
**Tags:** driveos-dl\
**Created:** [February 26, 2024, 11:49am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056 "2024-02-26T11:49:05Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![gokul.soman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@gokul.soman](https://forums.developer.nvidia.com/u/gokul.soman)\
**Post date:** [February 26, 2024, 11:49am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/1 "2024-02-26T11:49:05Z")

</div>

Please provide the following info (tick the boxes after creating this topic):  
**Software Version**  
 DRIVE OS 6.0.8.1  
[\*] DRIVE OS 6.0.6  
 DRIVE OS 6.0.5  
 DRIVE OS 6.0.4 (rev. 1)  
 DRIVE OS 6.0.4 SDK  
 other

**Target Operating System**  
[\*] Linux  
 QNX  
 other

**Hardware Platform**  
 DRIVE AGX Orin Developer Kit (940-63710-0010-300)  
 DRIVE AGX Orin Developer Kit (940-63710-0010-200)  
 DRIVE AGX Orin Developer Kit (940-63710-0010-100)  
 DRIVE AGX Orin Developer Kit (940-63710-0010-D00)  
 DRIVE AGX Orin Developer Kit (940-63710-0010-C00)  
[\*] DRIVE AGX Orin Developer Kit (not sure its number)  
 other

**SDK Manager Version**  
 1.9.3.10904  
[\*] other, 2.0.0

**Host Machine Version**  
[\*] native Ubuntu Linux 20.04 Host installed with SDK Manager  
 native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers  
 native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers  
 other

Hi,  
I am trying to run the face detector model (blazeface model , used by mediapipe) [blaze\_face\_short\_range.zip](https://forums.developer.nvidia.com/uploads/short-url/bmZcpyX4M2EIK00Uym4VlO97Qx.zip) (248.1 KB)

I am trying to modify the code so that the detections happen. I am following implementation from [here](https://github.com/longnguyen2/blazeface/blob/master/blazeface.py#L298) to get the preprocessing and postprocessing steps.

For the preprocessing, I used  
[blaze\_face\_short\_range.trt.json.txt](https://forums.developer.nvidia.com/uploads/short-url/wNaKKO7Vd4CfE8SEFPzlDwwnBwJ.txt) (350 Bytes)

Also, attaching the tensorrt model I obtained after compiling using trtexec (the compilation failed with ./TensorRTOptimizationTool.  
[blaze\_face\_short\_range.zip](https://forums.developer.nvidia.com/uploads/short-url/dF9RO1mBgzAy91vQA0gfHPF29sQ.zip) (331.5 KB)

The post processing steps, I followed from the github implementation I had shared above.  
The code I am using is a modification of sample object detector tracker.  
The code is running, and I am printing out the box sizes, and confidence scores.  
The confidence scores are coming low, so I am using a lower threshold.  
The boxes are also coming at lower sizes.

I tried disabling the feature tracker as well.

Attaching the code here  
[main.zip](https://forums.developer.nvidia.com/uploads/short-url/nS8vtfb7v86omBjmW82wBzeudK9.zip) (9.8 KB)

Thanks,  
Gokul

---

<div class="post-metadata">

**Author:** ![SivaRamaKrishnaNV](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/sivaramakrishnanv/32/14043_2.png) [@SivaRamaKrishnaNV](https://forums.developer.nvidia.com/u/SivaRamaKrishnaNV)\
**Post date:** [February 26, 2024, 4:54pm UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/3 "2024-02-26T16:54:50Z")

</div>

> [@gokul.soman](#):
>
> the compilation failed with ./TensorRTOptimizationTool.

Dear @gokul.soman,  
You need to use the model generated using tensorRT\_Optimization tool to integrate your model into DW DNN framework. Could you share the used command and full log.

---

<div class="post-metadata">

**Author:** ![gokul.soman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@gokul.soman](https://forums.developer.nvidia.com/u/gokul.soman)\
**Post date:** [February 27, 2024, 6:04am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/4 "2024-02-27T06:04:23Z")

</div>

Hi,

this is the log when using tensorRT\_optimization tool

> nvidia@tegra-ubuntu:/usr/local/driveworks/tools/dnn$ sudo ./tensorRT\_optimization --onnxFile=/home/nvidia/gokul/blaze\_face\_short\_range.onnx --out=~/gokul/optimized/face\_detector\_trtot.bin --modelType=onnx  
> [27-02-2024 06:01:51] WARNING: ExplicitBatch is enabled by default for ONNX models.  
> [27-02-2024 06:01:51] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3  
> Use model.batch[BATCH\_SIZE].onnx (eg. `model.batch8.onnx` with ONNX batch size 8) with accompanying batch\_sizes in configuration.json instead.  
> [27-02-2024 06:01:52] DNNGenerator: Initializing TensorRT generation on model /home/nvidia/gokul/blaze\_face\_short\_range.onnx.  
> [27-02-2024 06:01:52] onnx2trt\_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.  
> [27-02-2024 06:01:52] DNNGenerator: Input “input”: 1x3x128x128  
> [27-02-2024 06:01:52] DNNGenerator: Output “regressors”: 1x896x16  
> [27-02-2024 06:01:52] DNNGenerator: Output “classificators”: 1x896x1  
> [27-02-2024 06:02:15] Releasing Driveworks SDK Context  
> Error: DW\_FILE\_INVALID: Cannot open file for writing. Please ensure that you have write permissions.

---

<div class="post-metadata">

**Author:** ![SivaRamaKrishnaNV](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/sivaramakrishnanv/32/14043_2.png) [@SivaRamaKrishnaNV](https://forums.developer.nvidia.com/u/SivaRamaKrishnaNV)\
**Post date:** [February 27, 2024, 6:39am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/5 "2024-02-27T06:39:51Z")

</div>

> [@gokul.soman](#):
>
> Error: DW\_FILE\_INVALID: Cannot open file for writing. Please ensure that you have write permissions.

Dear @gokul.soman,  
Looks like some issue with file permission. Could you check running from home folder with out sudo. Also, make sure the output folder is empty and has write permission.

Also, any reason for using DRIVE OS 6.0.6 and not latest release?

---

<div class="post-metadata">

**Author:** ![SivaRamaKrishnaNV](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/sivaramakrishnanv/32/14043_2.png) [@SivaRamaKrishnaNV](https://forums.developer.nvidia.com/u/SivaRamaKrishnaNV)\
**Post date:** [February 27, 2024, 7:31am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/6 "2024-02-27T07:31:25Z")

</div>

Dear @gokul.soman,  
I quickly tested the model on DRIVE OS 6.0.8.1 release. I don’t see any errors as such

```auto
nvidia@tegra-ubuntu:~/siva$ /usr/local/driveworks/tools/dnn/tensorRT_optimization --onnxFile=/home/nvidia/siva/blaze_face_short_range.onnx --out=/home/nvidia/siva/blaze.bin
ProgramArguments: Missing argument 'outputBlobs' requested
ProgramArguments: Missing argument 'inputBlobs' requested
ProgramArguments: Missing argument 'inputDims' requested
[27-02-2024 07:28:13] WARNING: ExplicitBatch is enabled by default for ONNX models.
[27-02-2024 07:28:13] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3
                      Use model.batch[BATCH_SIZE].onnx (eg. `model.batch8.onnx` with ONNX batch size 8) with accompanying batch_sizes in configuration.json instead.
[27-02-2024 07:28:15] DNNGenerator: Initializing TensorRT generation on model /home/nvidia/siva/blaze_face_short_range.onnx.
[27-02-2024 07:28:16] onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[27-02-2024 07:28:16] DNNGenerator: Input "input": 1x3x128x128
[27-02-2024 07:28:16] DNNGenerator: Output "regressors": 1x896x16
[27-02-2024 07:28:16] DNNGenerator: Output "classificators": 1x896x1
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 0: 2.965376 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 1: 0.391840 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 2: 0.392512 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 3: 0.395680 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 4: 0.423296 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 5: 0.392576 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 6: 0.391072 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 7: 0.390976 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 8: 0.387328 ms.
[27-02-2024 07:28:43] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:28:43] DNNValidator: Iteration 9: 0.393952 ms.
[27-02-2024 07:28:43] DNNValidator: Average over 10 runs is 0.652461 ms.
[27-02-2024 07:28:43] [27-02-2024 07:28:43] Releasing Driveworks SDK Context

```

---

<div class="post-metadata">

**Author:** ![gokul.soman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@gokul.soman](https://forums.developer.nvidia.com/u/gokul.soman)\
**Post date:** [February 27, 2024, 7:37am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/7 "2024-02-27T07:37:54Z")

</div>

Hi @SivaRamaKrishnaNV ,  
I moved the binary into a folder in home directory and ran the same. but then again getting different error

> nvidia@tegra-ubuntu:~/gokul$ ./tensorRT\_optimization --modelType=onnx --onnxFile=blaze\_face\_short\_range.onnx --out=test\_folder/test.bin  
> [27-02-2024 07:33:52] WARNING: ExplicitBatch is enabled by default for ONNX models.  
> [27-02-2024 07:33:52] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3  
> Use model.batch[BATCH\_SIZE].onnx (eg. `model.batch8.onnx` with ONNX batch size 8) with accompanying batch\_sizes in configuration.json instead.  
> [27-02-2024 07:34:05] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars)  
> [27-02-2024 07:34:05] DNNGenerator: Initializing TensorRT generation on model blaze\_face\_short\_range.onnx.  
> [27-02-2024 07:34:05] onnx2trt\_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.  
> [27-02-2024 07:34:05] DNNGenerator: Input “input”: 1x3x128x128  
> [27-02-2024 07:34:05] DNNGenerator: Output “regressors”: 1x896x16  
> [27-02-2024 07:34:05] DNNGenerator: Output “classificators”: 1x896x1  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:06] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:07] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:08] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:09] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:10] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:11] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:12] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:13] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:14] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:15] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:16] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] Caught cuda error during timing, format rejected.  
> [27-02-2024 07:34:17] GPU error during getBestTactic: conv2d + activation : no kernel image is available for execution on the device  
> [27-02-2024 07:34:17] 1: [convBaseRunner.cpp::execute::271] Error Code 1: Cask (Cask convolution execution)  
> [27-02-2024 07:34:17] 1: [checkMacros.cpp::catchCudaError::202] Error Code 1: Cuda Runtime (no kernel image is available for execution on the device)  
> [27-02-2024 07:34:18] 2: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )  
> [27-02-2024 07:34:18] Releasing Driveworks SDK Context  
> Error: DW\_INTERNAL\_ERROR: DNNGenerator: Network build and serialization failed.

So, is the solution to flash newer version of driveOS?

Thanks,  
Gokul

---

<div class="post-metadata">

**Author:** ![SivaRamaKrishnaNV](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/sivaramakrishnanv/32/14043_2.png) [@SivaRamaKrishnaNV](https://forums.developer.nvidia.com/u/SivaRamaKrishnaNV)\
**Post date:** [February 27, 2024, 7:47am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/8 "2024-02-27T07:47:32Z")

</div>

> [@gokul.soman](#):
>
> So, is the solution to flash newer version of driveOS?

I copied the binary to new folder and tested. I don’t see any issue with model generation.

```auto
nvidia@tegra-ubuntu:~/siva$ cp /usr/local/driveworks/tools/dnn/tensorRT_optimization .
nvidia@tegra-ubuntu:~/siva$ ./tensorRT_optimization --onnxFile= blaze
blaze.bin blaze_face_short_range.onnx
nvidia@tegra-ubuntu:~/siva$ ./tensorRT_optimization --onnxFile=blaze_face_short_range.onnx --out=blaze-new.bin
ProgramArguments: Missing argument 'outputBlobs' requested
ProgramArguments: Missing argument 'inputBlobs' requested
ProgramArguments: Missing argument 'inputDims' requested
[27-02-2024 07:44:27] WARNING: ExplicitBatch is enabled by default for ONNX models.
[27-02-2024 07:44:27] WARNING: --batchSize is ignored for explicit batch ONNX models for TRT versions after 6.2.0.3
                      Use model.batch[BATCH_SIZE].onnx (eg. `model.batch8.onnx` with ONNX batch size 8) with accompanying batch_sizes in configuration.json instead.
[27-02-2024 07:44:30] DNNGenerator: Initializing TensorRT generation on model blaze_face_short_range.onnx.
[27-02-2024 07:44:30] onnx2trt_utils.cpp:372: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[27-02-2024 07:44:30] DNNGenerator: Input "input": 1x3x128x128
[27-02-2024 07:44:30] DNNGenerator: Output "regressors": 1x896x16
[27-02-2024 07:44:30] DNNGenerator: Output "classificators": 1x896x1
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 0: 2.996320 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 1: 0.413728 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 2: 0.400288 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 3: 0.440544 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 4: 0.386688 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 5: 0.388768 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 6: 0.395232 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 7: 0.385344 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 8: 0.383104 ms.
[27-02-2024 07:44:56] GraphRecorder: CUDA graph disabled, skip recording
[27-02-2024 07:44:56] DNNValidator: Iteration 9: 0.382976 ms.
[27-02-2024 07:44:56] DNNValidator: Average over 10 runs is 0.657299 ms.
[27-02-2024 07:44:56] [27-02-2024 07:44:56] Releasing Driveworks SDK Context

```

I am not sure if the issue related to your environment. We recommend to upgrade to new release if you don’t have any dependency on DRIVE OS version.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [March 21, 2024, 7:19am UTC](https://forums.developer.nvidia.com/t/modified-sample-object-detector-tracker/284056/10 "2024-03-21T07:19:34Z")

</div>

Closed on March 21, 2024, 7:19am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
