Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError

imzhuo · August 22, 2019, 7:26am

basic information

System: Linux(ubuntu16.04)
GPU: NVIDIA RTX 2080
CUDA: cuda-9.0, cudnn7.4.2

problem description:

cuda-memcheck my executable gives this information:
Program hit cudaErrorInvalidValue (error 11) due to “invalid argument” on CUDA API call to cudaFuncSetAttribute.

deviceQuery result:

./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: “GeForce RTX 2080”
CUDA Driver Version / Runtime Version 10.1 / 9.0
CUDA Capability Major/Minor version number: 7.5
Total amount of global memory: 7951 MBytes (8337227776 bytes)
MapSMtoCores for SM 7.5 is undefined. Default to use 64 Cores/SM
MapSMtoCores for SM 7.5 is undefined. Default to use 64 Cores/SM
(46) Multiprocessors, ( 64) CUDA Cores/MP: 2944 CUDA Cores
GPU Max Clock rate: 1815 MHz (1.81 GHz)
Memory Clock rate: 7000 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 4194304 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 1024
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 3 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 9.0, NumDevs = 1
Result = PASS

problem description

I’m trying to poring my project(cuBlas/CuDNN based), from Windows 7 to Linux(ubuntu 16.04).

My code run OK on Windows but failed on Linux. After googling around, I use cuda-memcheck to figure out what the bug is. The minimal code to reproduce the error (as the title) is:

#include <cuda.h>
#include <cublas_v2.h>
#include <cudnn.h>

                                                           
#include <stdio.h>
#include <iostream>

using namespace std;



#define cudaDrvCheck(call) \
{ \
    const CUresult state = call; \
    if ((state) != cudaSuccess) { \
        fprintf(stderr, "CUDA Driver Error: %s %s %d\n",                     getCudaDrvErrorString(state), __FILE__, __LINE__); \
        exit(1); \
        } \
}

#define cudaErrCheck(condition) \
    do { \
        cudaError_t error = condition; \
        if (error != cudaSuccess) { \
            fprintf(stderr, "CUDA Error: %s %s %d\n",                        cudaGetErrorString(error), __FILE__, __LINE__); \
            exit(-1); \
                        } \
            } while (0)

#define cublasErrCheck(condition) \
    do { \
        cublasStatus_t status = condition; \
        if (status != CUBLAS_STATUS_SUCCESS) { \
            fprintf(stderr, "cublas Error: error code : %d %s %d\n", status, __FILE__, __LINE__); \
            exit(-1); \
                        } \
            } while (0)

#define cudnnErrCheck(condition) \
    do { \
        cudnnStatus_t status = condition; \
        if (status != CUDNN_STATUS_SUCCESS) { \
            fprintf(stderr, "cuDNN Error: %s %s %d\n",                       cudnnGetErrorString(status), __FILE__, __LINE__); \
            exit(-1); \
                        } \
            } while (0)




typedef struct RockEngine {
    cublasHandle_t handle_cublas;
    cudnnHandle_t handle_cudnn;
} RockEngine;

int main() {

    RockEngine* engine = (RockEngine*)malloc(sizeof(RockEngine));

    cudaError_t cuda_status = cudaSetDevice(0);
    cudaErrCheck(cuda_status);

    int gpu_id=0;
    cudaErrCheck(cudaGetDevice(&gpu_id));
    cout << "!!! gpu_id is " << gpu_id << endl;

    cudnnStatus_t cudnn_status = cudnnCreate(&engine->handle_cudnn);
    cudnnErrCheck(cudnn_status);

    cublasStatus_t cublas_status = cublasCreate(&engine->handle_cublas);
    cublasErrCheck(cublas_status);


    return 0;
}

What do I execute
After compilation, the executable I get is called “rock”.

What I’m running is:

cuda-memcheck rock  > log.txt 2>&1

The whole log
on my machine (whose information is listed in the very begin of this question), the log.txt is:

!!! gpu_id is 0

I! CuDNN (v7402) function cudnnCreate() called:
i! Time: 2019-08-22T15:24:10.299850 (0d+0h+0m+0s since start)
i! Process=768; Thread=768; GPU=NULL; Handle=NULL; StreamId=NULL.

========= CUDA-MEMCHECK
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x157f001]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13ab8be]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x13abc20]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 [0x104d928]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcudnn.so.7 (cudnnCreate + 0x5b5) [0x95705]
=========     Host Frame:rock [0x3690]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaFuncSetAttribute. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x390523]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x405871]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x3992f5]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 [0x399620]
=========     Host Frame:/usr/local/cuda-9.0/lib64/libcublas.so.9.0 (cublasCreate_v2 + 0x3e8) [0x79358]
=========     Host Frame:rock [0x36e6]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xf0) [0x20830]
=========     Host Frame:rock [0x34b9]
=========
========= ERROR SUMMARY: 36 errors

However, on my another machine, which is ubuntu16.04 with GTX 1080Ti GPU, cuda-9.0, the log is without any error. I wonder why this error occur, and how can I resolve it? Thanks for any suggestions.

Robert_Crovella · August 22, 2019, 1:52pm

try a newer cudnn version

imzhuo · August 23, 2019, 1:45am

@Robert_Crovella Thanks for your suggestion. When I remove the usage of cuDNN in my code, and also remove the linked cuDNN library option in my CMakeListst.txt， cuda-memcheck still gives the same error. That error is about cublasCreate.

I’ve also tested under CUDA10.0 environment. And the result is the same.

What I’m curious is that when I pip install torch, its torch.version.cuda is 10.0.130, however, this version of pytorch can use GPU.

imzhuo · August 23, 2019, 12:45pm

What I’ve tested out is that:

for RTX 2080,

CUDA9.0 works, cuBLAS fails, cuDNN fail;
CUDA10.0 works, cuBLAS fails, cuDNN works;
CUDA10.1 works, cuBLAS works, cuDNN works.

Thus, I can only use CUDA10.1 on my GTX 2080 GPU to ensure my code run OK.

Robert_Crovella · August 23, 2019, 3:24pm

CUDA 10.0 had a bug with CUBLAS in cublasCreate that would report the invalid argument error from internal API usage.

This was fixed in cuda 10.1

AFAIK the cublasCreate function in CUDA 10.0 didn’t actually report an error - and CUBLAS was still functional. This only showed up when running under cuda-memcheck.

Topic		Replies	Views
Is my card burned? CUDA Setup and Installation	15	2645	June 27, 2013
cufftPlan1D initialisation hides subsequence memory access errors CUDA Programming and Performance	8	653	November 24, 2020
[SOLVED]memory leak issues CUDA Programming and Performance	12	3606	October 12, 2015
cuda-memcheck errors from SDK samples - ignore? OptiX	3	1157	June 14, 2022
Got error “Address 0x4068 is out of bounds”, when matrix is large when using cusparseSpMV GPU-Accelerated Libraries cuda , cusparse	4	175	June 17, 2024
Error when running cuda-memcheck on command line. CUDA-MEMCHECK	3	2546	December 4, 2017
Unable to run several CUDA samples. CUDA Programming and Performance	2	821	April 1, 2019
The result of device code is wrong when without a "printf" CUDA Programming and Performance	8	2006	October 22, 2016
Thrust error with cuda 9.1 with windows 10 CUDA Programming and Performance	0	1009	April 10, 2018
runtime error with cuda 9.1 CUDA Programming and Performance	18	4134	April 24, 2018

Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaGetLastError

Related topics