DW_CUDA_ERROR on DriveWork SDK v1.5.3584

Hello,

I have installed the new DriveWork SDK from NVidia and wanted to run the sample_object_detector. When doing so (on Ubuntu Computer) the following error occured:

Platform: Detected Generic x86 Platform
Initialize DriveWorks SDK v1.5.3584
Release build with GNU 4.9.4 from heads/buildbrain-branch-0-g38b3a75
TimeSource: monotonic epoch time offset is 1550192108463707

Driveworks exception thrown: DW_CUDA_ERROR: Platform: cannot acquire cuda context : unkown error

terminate called after throwing an instance of 'std::runtime_error'
 what(): [] DW Error DW_CUDA_ERROR executing DW function: 
 dwInitialize(&m_sdk, DW_VERSION, &SDKParams)
 at /dvs/git/dirty/gitlab-master_av/dw/sdk/samples/dnn/sample_object_detector/main.cpp:135
Aborted (core dumped)

CUDA 10.0 is installed and the installed GPU Driver Version is: 410.78.
Does anyone know a solution to this Problem?

Thank you

Dear HV_ZF,
can you check ldd of sample_object_detector to confirm if it gets linked to right CUDA so files. If not please set LD_LIBRARY_PATH accordingly.

Hello SivaRamaKrishna,

so here is the update:

/usr/local/driveworks-1.5/bin$ ldd ./sample_object_detector 
	linux-vdso.so.1 =>  (0x00007ffc0edb6000)
	libdriveworks.so.1 => /usr/local/driveworks-1.5/bin/./../lib/libdriveworks.so.1 (0x00007f033fa63000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f033f85b000)
	libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f033f521000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f033f304000)
	libXrandr.so.2 => /usr/lib/x86_64-linux-gnu/libXrandr.so.2 (0x00007f033f0f9000)
	libXinerama.so.1 => /usr/lib/x86_64-linux-gnu/libXinerama.so.1 (0x00007f033eef6000)
	libXxf86vm.so.1 => /usr/lib/x86_64-linux-gnu/libXxf86vm.so.1 (0x00007f033ecf0000)
	libXcursor.so.1 => /usr/lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007f033eae6000)
	libGL.so.1 => /usr/lib/nvidia-410/libGL.so.1 (0x00007f033e7a8000)
	libcudart.so.10.0 => /usr/local/cuda/lib64/libcudart.so.10.0 (0x00007f033e52e000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f033e32a000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f033dfa8000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f033dc9f000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f033da89000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f033d6bf000)
	libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f033c5bf000)
	libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f033c3a7000)
	libnvcuvid.so.1 => /usr/lib/nvidia-410/libnvcuvid.so.1 (0x00007f033bef7000)
	libnvinfer.so.4 => /usr/local/driveworks-1.5/bin/./../lib/../lib/libnvinfer.so.4 (0x00007f0335f29000)
	libnvparsers.so.4.1.4 => /usr/local/driveworks-1.5/bin/./../lib/../lib/libnvparsers.so.4.1.4 (0x00007f0335963000)
	libGLU.so.1 => /usr/lib/x86_64-linux-gnu/libGLU.so.1 (0x00007f03356f4000)
	libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f03354d2000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0340bea000)
	libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f03352b0000)
	libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f033509e000)
	libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f0334e94000)
	libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f0334c8e000)
	libnvidia-tls.so.410.78 => /usr/lib/nvidia-410/tls/libnvidia-tls.so.410.78 (0x00007f0334a8a000)
	libnvidia-glcore.so.410.78 => /usr/lib/nvidia-410/libnvidia-glcore.so.410.78 (0x00007f0332eaf000)
	libnvidia-fatbinaryloader.so.410.78 => /usr/lib/nvidia-410/libnvidia-fatbinaryloader.so.410.78 (0x00007f0332c62000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f0340d99000)
	libcudnn.so.7 => /usr/local/driveworks-1.5/bin/./../lib/../lib/libcudnn.so.7 (0x00007f031f0de000)
	libcublas.so.10.0 => /usr/local/cuda/lib64/libcublas.so.10.0 (0x00007f031ab44000)
	libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f031a940000)
	libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f031a73a000)

so it is linked to the CUDA, I assume

now when I run again:

/usr/local/driveworks-1.5/bin$ ./sample_object_detector 
[26-02-2019 10:34:26] Platform: Detected Generic x86 Platform
[26-02-2019 10:34:26] Initialize DriveWorks SDK v1.5.3584
[26-02-2019 10:34:26] Release build with GNU 4.9.4 from heads/buildbrain-branch-0-g38b3a75
[26-02-2019 10:34:26] TimeSource: monotonic epoch time offset is 1551171285306328
[26-02-2019 10:34:26] Platform: number of GPU devices detected 1
[26-02-2019 10:34:26] Platform: currently selected GPU device discrete ID 0
[26-02-2019 10:34:26] SDK: Resources mounted from .././data/resources
[26-02-2019 10:34:26] SensorFactory::createSensor() -> camera.virtual, profiling=1,tensorRT_model=,video=.././data/samples/sfm/triangulation/video_0.h264
[26-02-2019 10:34:26] CameraNVCUVID: no seek table found at .././data/samples/sfm/triangulation/video_0.h264.seek, seeking is not available.
SimpleCamera: Camera image: 1280x800
Camera image with 1280x800 at 30 FPS
[26-02-2019 10:34:26] Glob Size is 46033424 bytes.
[26-02-2019 10:34:26] Added linear block of size 30670848
[26-02-2019 10:34:26] Added linear block of size 23003136
[26-02-2019 10:34:26] Added linear block of size 7667712
[26-02-2019 10:34:26] Added linear block of size 7667712
sample_object_detector: caskConvolutionLayer.cpp:145: virtual void nvinfer1::task::caskConvolutionLayer::allocateResources(const nvinfer1::cudnn::CommonContext&): Assertion `configIsValid(context)' failed.
Aborted (core dumped)

I still receive an error

Dear HV_ZF,
This is strange. Can you share output of CUDA deviceQuery sample(/usr/local/cuda-10.0/samples/1_utilities/deviceQuery)

Dear SivaRamaKrishna,

here is the output of the deviceQuery

/usr/local/cuda-10.0/samples/bin/x86_64/linux/release$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Quadro M2000M"
  CUDA Driver Version / Runtime Version          10.0 / 10.0
  CUDA Capability Major/Minor version number:    5.0
  Total amount of global memory:                 4042 MBytes (4238802944 bytes)
  ( 5) Multiprocessors, (128) CUDA Cores/MP:     640 CUDA Cores
  GPU Max Clock rate:                            1137 MHz (1.14 GHz)
  Memory Clock rate:                             2505 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 2097152 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.0, CUDA Runtime Version = 10.0, NumDevs = 1
Result = PASS

Dear HV_ZF,
It seems you are having maxwell based GPU. But minimim pascal based GPU is recommended to run driveworks samples on host. I think that is the reason. Let us know if you notice this issue with pascal based cards.

Dear SivaRamaKrishna,

Thank you for your help!
That pascal is the minimum required architecture was not written in the release notes for the latest DriveWorks. Maybe it would be a good idea to add this (I only found the minimum requirements, when I was searching, in the Drive_Software_Docs).
In the Release Notes for DriveWorks1.2 from last October the minimum requirements were given (mininum was Maxwell) and thus I assumed that they didn’t changed, because it was not mentioned.

To test it on a pascal based card, is currently difficult because I do not have access to one.

Now I could try it on a newer architecture and then it works, thus that was the problem!

Thanks again for your help!