How would I make kernelExecTimeout Shorter?

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Suppose I explicitly want to limit my kernels to 0.5 seconds and prefer buggy runaway GPU threads result in killing the whole process. How would I reduce the OS/Driver timeout?

Running the common Jetpack install of Ubuntu on Xavier.

3

hello cory.bloyd,

may I know what’s the use-case, or, which application you’re running with.
thanks

5

I’m working on a system composed of multiple processes written by multiple teams. Each step is explicitly small and no one expects to run a single kernel for as long as an entire second. But, bugs happen. If there is a runaway loop in a kernel, we’d rather not wait 5 seconds to start over.

6

hello cory.bloyd,

it’s related to CUDA scheduler. I’m afraid you cannot adjust timeout values, kernelExecTimeout.
you may have implementation to terminate your process while there’s a failure,
thanks

1 Like
9

Hi, cory.bloyd

Do you find a way to solve this problem?
If not, could you file a new topic with a source code to reproduce the long kernel execution time issue?

Thanks.