Cutensor produces runtime error on linux, but not windows

thothgunter · September 27, 2021, 4:02am

Hello,
I rewrote the yolov4 directly using the cuda libs and it worked on windows. However when compiling on linux(Ubuntu 18.04) I receive runtime errors executing the function cutensorElementwiseBinary. The error code is 15, “CUTENSOR_STATUS_NOT_SUPPORTED”. Upon investigation it seems like cutensorInitTensorDescriptor filled out the descriptor differently on the two operating systems. I am using cuda 11 on both and cutensor version 1.3.3.
The code can be found here: GitHub - TKGgunter/yolov4_tiny_rs: A rust implementation of yolov4_tiny algorithm. .

Thanks for any help.

pspringer · September 27, 2021, 1:09pm

Would you be able to isolate the failing call s.t. we could have a closer look?

cuTENSOR’s logging capabilities (see User Guide — cuTENSOR 1.6.0 documentation) might be useful to you; please see if CUTENSOR_LOG_LEVEL=1 gives you more insights. If not, could you please report the output of CUTENSOR_LOG_LEVEL=5?

thothgunter · September 27, 2021, 1:50pm

I’ll write a smaller standalone program later, but the relevant code links are below. The first link is to the first instance in main where the error is returned. The second is the source code of that function. The output of CUTENSOR_LOG_LEVEL=5 is given below.

github.com

TKGgunter/yolov4_tiny_rs/blob/fa6b4efd0b98472ab73aba5821d6613c078e6b5e/src/main.rs#L806


      
            mbbox_output.data.offset(5*4),
            (&0f32 as *const _) as *const _,
            prob_desc26,
            prob_ttensor_26.data,
          ));
          
          {
            //pred_xy = STRIDES[i] * ( tf.sigmoid(conv_raw_dxdy) * XYSCALE[i] - 0.5 * 
            //(XYSCALE[i] - 1) + xy_grid)
          
              calc_elementwise_binary(&cutensor_handle, cuda_stream, &dim_labels, &dxdy_ttensor_13, 
                                      &mut dxdy_ttensor_ones_13, Some(&mut dxdy_ttensor_out_13), GpuTensorOps::Add,
                                      1.05, 1f32);
          
          
              calc_elementwise_binary(&cutensor_handle, cuda_stream, &dim_labels, 
                                      &GpuTensorTensor{ descriptor: dxdy_ttensor_out_13.descriptor, data: xygrid_13_data, .. GpuTensorTensor::default()}, //TODO should set this up above
                                      &mut dxdy_ttensor_out_13, 
                                      None, GpuTensorOps::Add,
                                      32f32, 32f32);

github.com

TKGgunter/yolov4_tiny_rs/blob/fa6b4efd0b98472ab73aba5821d6613c078e6b5e/src/tensor.rs#L124


      
              //pub const cutensorOperator_t_CUTENSOR_OP_CEIL: cutensorOperator_t = 37;
              //pub const cutensorOperator_t_CUTENSOR_OP_FLOOR: cutensorOperator_t = 38;
              Add = cutensorOperator_t_CUTENSOR_OP_ADD as isize,
              Mul = cutensorOperator_t_CUTENSOR_OP_MUL as isize,
              //pub const cutensorOperator_t_CUTENSOR_OP_MAX: cutensorOperator_t = 6;
              //pub const cutensorOperator_t_CUTENSOR_OP_MIN: cutensorOperator_t = 7;
              //pub const cutensorOperator_t_CUTENSOR_OP_UNKNOWN: cutensorOperator_t = 126;
          }
          
          //TODO document the behavior of this function. It will not be clear to new users 
          pub fn calc_elementwise_binary(cutensor_handle: &cutensorHandle_t, cuda_stream: cudaStream_t, dim_labels: &[char], x1: &GpuTensorTensor, x2: &mut GpuTensorTensor, _y: Option<&mut GpuTensorTensor>, op: GpuTensorOps, alpha: f32, beta: f32){
          
              let y_data = match _y {
                  Some(a)=>{ a.data },
                  _=>{ x2.data }
              };
              tensor_error!(cutensorElementwiseBinary(
                  cutensor_handle,
          
                  (&alpha as *const f32) as _,
                  x1.data,

[2021-09-27 08:35:48][cuTENSOR][24264][Api][cutensorInitTensorDescriptor] handle=0X7FFCBB67A9D0, desc_=0X7FFCBB6D2D98, numModes=4, extent=0X560BC4D143F0, stride=0X0, dataType=0, op=11
[2021-09-27 08:35:48][cuTENSOR][24264][Api][cutensorInitTensorDescriptor] handle=0X7FFCBB67A9D0, desc_=0X7FFCBB6D3000, numModes=4, extent=0X560BC4D143F0, stride=0X0, dataType=0, op=1

useA1 useB0 useC1
-Pcs -modeC97,98,99,100 -strideC1,13,169,507, -gamma0.700000 -opC1 -Pas -modeA97,98,99,100 -strideA1,13,169,507, -alpha1.300000 -opA11 -extent100=2,99=3,97=13,98=13, -opAB3 -opABC3 -Pcomps -Relementwise

thothgunter · September 27, 2021, 1:51pm

Just encase it is unclear the function of interest is

calc_elementwise_binary

thothgunter · September 27, 2021, 7:12pm

An example has been write and can be found here https://github.com/TKGgunter/yolov4_tiny_rs/blob/master/cuda11-cutensor-sys/examples/linux_error.rs.
To run down load the contents of https://github.com/TKGgunter/yolov4_tiny_rs/tree/master/cuda11-cutensor-sys. and run the command

cargo run --example linux_error

thothgunter · September 29, 2021, 6:35pm

A C/C++ version can be found here yolov4_tiny_rs/linux_error.c at master · TKGgunter/yolov4_tiny_rs · GitHub

thothgunter · October 21, 2021, 2:50pm

Hello @pspringer , have you or someone at Nvidia been able to look into this problem? Do are there any solutions/work a rounds to this issue?

Topic		Replies	Views
Compute sanitizer return error code 500 GPU-Accelerated Libraries cutensor	2	743	September 29, 2022
cuTENSOR internal error GPU-Accelerated Libraries cuda , cutensor	2	934	August 8, 2022
cuTENSOR internal error CUDA Setup and Installation cuda	1	739	August 8, 2022
cuTensor Error in executeCutensor: 18 (Internal cuTensor reformat failed) TensorRT	3	2293	December 17, 2019
Just Released: cuTENSOR V1.5 Technical Blog	0	237	June 21, 2022
Wrong results when using input tensor as output tensor for cuTENSOR GPU-Accelerated Libraries cublas , cutensor	1	479	November 20, 2023
cuTENSOR internal error on Jetson AGX Xavier Jetson Xavier NX cuda	2	354	August 9, 2022
cuTENSOR 2.0: A Comprehensive Guide for Accelerating Tensor Computations Technical Blog	1	298	March 10, 2024
Deepstream_lpr_app for DS6 - libcufft.so.10: cannot open shared object TAO Toolkit	17	2590	May 11, 2022
Tesla T4: Cudnn Error in execute: 8 (CUDNN_STATUS_EXECUTION_FAILED) TensorRT	2	1565	July 17, 2020

Cutensor produces runtime error on linux, but not windows

Related topics