Performance regression for vkCreateDevice with GSP firmware enabled

I have noticed this for a while and that is that certain functions like vkCreateDevice takes much longer time with GSP on vs GSP off.

I have made a little test program to show the difference: ( change extension to .c )
vulkan_setup_test.txt (10.8 KB)

and compile with: gcc -o vulkan_setup_test vulkan_setup_test.c -lvulkan -lSDL2 -I/usr/include/SDL2

This is my result on rtx 3080 and latest 570.124.04 under gnome/wayland:
With GSP on:
Instance Creation: 4.1752 ms
Surface Creation: 0.1254 ms
Physical Device Selection: 0.1051 ms
Logical Device Creation: 72.4314 ms
Total Setup Time: 76.8372 ms

With GSP off:
Instance Creation: 1.8936 ms
Surface Creation: 0.1254 ms
Physical Device Selection: 0.1394 ms
Logical Device Creation: 21.7755 ms
Total Setup Time: 23.9339 ms

There is about 50ms diff here, which is mostly the vkCreateDevice function and its repeatable on all drivers , open kernel modules or not as long as one is using GSP firmware the penalty is there.

( ps! this is not about stuttering with use of GSP its just about startup times, altough they might be related ).