# The calling process of \_\_device\_\_ function **URL:** \ **Category:** CUDA Programming and Performance\ **Tags:** cuda\ **Created:** [March 29, 2024, 6:11am UTC](https://forums.developer.nvidia.com/t/the-calling-process-of-device-function/287822 "2024-03-29T06:11:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![qwerty00](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@qwerty00](https://forums.developer.nvidia.com/u/qwerty00)\ **Post date:** [March 29, 2024, 6:11am UTC](https://forums.developer.nvidia.com/t/the-calling-process-of-device-function/287822/1 "2024-03-29T06:11:22Z")
I am curious about the calling details of a \_\_device\_\_ function. As is well-known, the call to a function in CPU should undergo the following steps: 1. The caller saves the context (registers). (This could also be implemented in the callee). 2. The caller pushes the parameters into a stack. 3. The callee executes the function. 4. The caller restores the context. This involves a stack. My question is that does cuda maintain such a stack for each thread? And does cuda save the context of each thread before calling a function? It could be quite space-consuming and slow (as the stack may reside in global or constant memory). ---
**Author:** ![njuffa](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@njuffa](https://forums.developer.nvidia.com/u/njuffa)\ **Post date:** [March 29, 2024, 6:58am UTC](https://forums.developer.nvidia.com/t/the-calling-process-of-device-function/287822/2 "2024-03-29T06:58:28Z")
You seem to describe older stack-based calling convention. Calling conventions for modern processor architectures (e.g. x86-64, AArch64) are typically mostly register centric. While NVIDIA has (to the best of my knowledge) _not publicly_ published an ABI for use on the GPU, it is clear from the support for various C++ features, support for separate compilation and device-code linking, and inspection of generated machine code that: (1) An ABI must exist, is documented internally, and used by tool chain components (2) Calling conventions governing function calls are largely register based For special situations, such as the passing of many function arguments, or function arguments or returns that have aggregate types, register-centric calling conventions still require a stack, and this applies to the ABI used on GPUs as well. A per-thread stack is implemented by CUDA. If you call `cudaDeviceGetLimit()` and inquired about `cudaLimitStackSize`, CUDA will report the current configured size of this stack. Function calls that use the internally specified ABI calling conventions are indeed pretty slow, which is why the CUDA toolchain heavily relies on function inlining as an optimization strategy. There is also support for link-time optimization that allows (among other things) function inlining across compilation units. I have never examined myself how well that works. The fact that separate compilation with link-time optimization often results in performance close to that achieved by code resulting from whole-program compilation would seem to indicate that this works quite well. ---
**Author:** ![qwerty00](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@qwerty00](https://forums.developer.nvidia.com/u/qwerty00)\ **Post date:** [March 29, 2024, 8:19am UTC](https://forums.developer.nvidia.com/t/the-calling-process-of-device-function/287822/3 "2024-03-29T08:19:39Z")
Thanks. I have a bit more questions. Though the parameter passing may be register-centric, the funtion being called will change the register environment in their running process. The question is: 1. Who is responsible for protecting the register context of the caller? 2. When compiling caller and callee seperately, the register usage of them is different. For example, the caller may use 80 registers while the callee use 100 registers, what will happen when calling the function? This also happens when different callers call the same callee. ---
**Author:** ![njuffa](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@njuffa](https://forums.developer.nvidia.com/u/njuffa)\ **Post date:** [March 29, 2024, 8:39am UTC](https://forums.developer.nvidia.com/t/the-calling-process-of-device-function/287822/4 "2024-03-29T08:39:31Z")
ABIs specify which registers are caller-saved and which ones are callee-saved. Since NVIDIA’s ABI for the GPU is not public (NVIDIA _may_ share it under NDA with partners in addition to using it internally, but I do _not_ know one way or the other), I do not have that information. You could reverse engineer most of it if you really needed it. What are you trying to accomplish that requires this information? I do not know how the device-side linker handles functions with different register requirements. I think I knew some basics at one time, but since I have not needed that information for 10+ years, I cannot recall anything about this topic. Why do you need to know? ---
**Author:** ![qwerty00](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@qwerty00](https://forums.developer.nvidia.com/u/qwerty00)\ **Post date:** [March 29, 2024, 12:08pm UTC](https://forums.developer.nvidia.com/t/the-calling-process-of-device-function/287822/5 "2024-03-29T12:08:02Z")
I’m just curious about how the function call process happens, because the register allocation scheme and performance requirement in GPU is totally different from that on CPU. I am wondering that if we use ABI to call a device function, the register occupancy may have some change and thus affect the performance? ---
**Author:** ![qwerty00](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@qwerty00](https://forums.developer.nvidia.com/u/qwerty00)\ **Post date:** [March 29, 2024, 2:27pm UTC](https://forums.developer.nvidia.com/t/the-calling-process-of-device-function/287822/6 "2024-03-29T14:27:41Z")
Here is a similar question. > [@Function call mechanism in CUDA](https://forums.developer.nvidia.com/t/function-call-mechanism-in-cuda/33380): > > Hello, I have been looking at the CUDA binary utilities of the online documentation as I wanted to know more about the actual ISA (out of curiosity). I have noticed there are several instructions to handle functions calls like JCAL, RET, EXIT, PRET… Where can I find more detailed information on these instructions? In particular I am interested in the the function call mechanism and how the function frames are handled, is there a “stack” like on x86 CPUs? Where are the return addresses stored…