Driveworks-2.2 samples crash on DriveAGX

I’ve cross-compiled the Driveworks-2.2 samples on my PC. When trying to run them on a DriveAGX Pegasus they crash a CUDA error 218 (cudaErrorInvalidPtx). It happens on both, the iGPU and the dGPU. Does anybody have an idea why this happens?

Hi klaus.kofler,

I haven’t seen anyone report this issue. I suspect your host build environment was messed up somehow.
You can try with pre-installed sample binaries under /usr/local/driveworks/bin to see if the same issue. Thanks!

Some of the samples in “/usr/local/driveworks/bin” work, some do not. For example “sample_autodrivingbeam” gives the error

The engine plan file is generated on an incompatible device, expecting compute 5.0 got compute 6.1, please rebuild.

After some research I found out that the CUDA error 218 is sometimes related to kernels being compiled with the wrong shader model. So in the file “/usr/local/driveworks-2.2/samples/cmake/SamplesConfiguration.cmake” I changed the line 25 from

set(CUDA_NVCC_FLAGS "-arch=sm_30;-lineinfo;-std=c++11;")

to

set(CUDA_NVCC_FLAGS "-gencode=arch=compute_72,code=sm_72 -gencode=arch=compute_75,code=sm_75;-lineinfo;-std=c++11;")

After that change, all the cross-compiled samples work on the DriveAGX. I did some testing, and it turns out, that when I cross compile, the DriveAGX can only run CUDA code compiled with sm_70 and sm_75 on the dGPU and only CUDA code compiled with sm_72 on the iGPU.

If I compile the same code on the DriveAGX, it can run all CUDA code compiled for a sm lower than the actual shader model of the corresponding GPU.

I cannot reproduce this issue. The default sample_autodrivingbeam on the target runs well on my side.

It looks the application ran on GPU with 5.0 compute capability instead of expected 7.2 (Xaiver) or 7.5 (dGPU) on the system.

Thanks for the feedback. We will try to make it more straight forward.

May I know running which cross compiled sample will hit the CUDA error 218 ? Thanks!

The pre-installed sample binaries in “/usr/local/driveworks/bin” are compiled for x86 on my system, so I ran and saw the error message on the host.

I see this error on all of the samples that do run a CUDA kernel. I can reproduce it with any simple CUDA program like for example the following:

#include <iostream>
#include <cuda.h>
__global__ void kernel()
{
        return;
}
int main()
{
        kernel<<<1,1>>>();
        auto err = cudaGetLastError();
        std::cout << "error: " << err << std::endl;
}

So I did a new installation of Drive10 on another System and I do not see this problem any more. I do not recall doing anything differently. Interestingly, I also see a different folder structure on the two installations: The second (working as expected) one has the folder “$DRIVE_SDK/drive-t186ref-linux/targetfs” while the first (not properly working) one has the two folders “$DRIVE_SDK/drive-t186ref-linux/targetfs_a” and “$DRIVE_SDK/drive-t186ref-linux/targetfs_b”. Maybe that hint could help figuring out what went wrong during the installation.

pre-installed sample binaries (/usr/local/driveworks/bin) on host were compilied for x86 and pre-installed sample binaries (/usr/local/driveworks/bin) on target were compiled for aarch64. Both are running well on my host and target respectively.

If you still want to clarify the problems, you can cross check with both your hosts.

On my setup, the pre-installed samples on the target are only present on Xavier A, not on B, hence I did not find them before. Not sure if that is normal though.

The pre-installed samples do work as expected. Same for programs compiled either directly on the DriveAGX or cross-compiled from my second host. Only the cross-compiled programs from my first host are problematic.

That’s correct. DriveWorks is only installed on Xavier A.