Fatal error: Unexpected cudaStreamQuery failure: unspecified launch failure

Hello! Friends,
I have Cuda 10.2 installed in ubuntu 18.04. Upon running GROMACS -a Molecular Dynamics software, if i run my simulations for shorter period - than the simulation runs smoothly on CPU, but if I run the simulations for longer period than simulation (part of it )runs on GPU (Quadro P620) and i get an error

Program: gmx mdrun, version 2020.3
Source file: src/gromacs/gpu_utils/cudautils.cuh (line 260)

Fatal error:
Unexpected cudaStreamQuery failure: unspecified launch failure

I have Nvidia Driver Version: 440.33.01 installed. I guess the problem is with CUDA and GPU . Can some one help me out with the problem

@Nvidia Admins, it has been over a year since post was created. Intel responds to their “Free” software questions within 48 hours and some of your organizational users are spending over $100K. Just saying “doesn’t look good”. Also if D.E. Shaw is interested in Molecular Dynamics, i.e. Schrödinger, it may be a good idea to look at this.

NVIDIA doesn’t have any forums for gromacs support. I suggest posting gromacs questions on a gromacs support forum.

I don’t imagine there are any gromacs experts here on this particular forum.

The only “CUDA” problem I know of that could manifest this way is a kernel timeout, which could be the case if you are driving a display from that P620 GPU. In that case, you don’t have many good options for running gromacs for heavy duty simulation on a display GPU. My suggestion then would be to run it on a GPU that is not driving the display.

Anything other than that would be a defect in gromacs (a guess).

The fact that the Quadro P620 is an older GPU at the very low end of NVIDIA’s offerings makes the time-out scenario mentioned by @Robert_Crovella quite likely, due to much longer kernel run times.

If you check the literature and the relevant GTC presentations you will notice that NVIDIA started working with all of the major molecular dynamic software providers more than a decade ago. This NVIDIA slide deck dated February 2017 provides an overview. NVIDIA’s job is to provide the respective developer teams with appropriate developer support for their GPU acceleration efforts. Supporting the products themselves is the job of the respective vendors or development teams.

Specifically for Gromacs: This team used an email list / archive for support for many years, but a few years ago they switched to an online forum. I have not checked it recently, but up to 2020 it used to be the case that questions there would be answered in a timely fashion by various core people in the Gromacs team, some of whom I have met personally and therefore recognize by name.