Address sanitizer bug with vkgetinstanceprocaddr

Vulkan API Version: 1.2.194
Vulkan Driver Version: 511.92.0

==22580==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x12a87a397f20 in thread T0
==22580==WARNING: Failed to use and restart external symbolizer!
#0 0x7ff922dad301 in _asan_wrap_GlobalSize+0x4a71d (C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x18004d301)
#1 0x7ff9b00af4e4 in RtlGetFullPathName_UstrEx+0x1c54 (C:\Windows\SYSTEM32\ntdll.dll+0x18001f4e4)
#2 0x7ff9b00b0b30 in RtlTryEnterCriticalSection+0x350 (C:\Windows\SYSTEM32\ntdll.dll+0x180020b30)
#3 0x7ff9b00bbed9 in RtlAllocateHeap+0x1539 (C:\Windows\SYSTEM32\ntdll.dll+0x18002bed9)
#4 0x7ff9b00bb3c6 in RtlAllocateHeap+0xa26 (C:\Windows\SYSTEM32\ntdll.dll+0x18002b3c6)
#5 0x7ff922dac47c in _asan_wrap_GlobalSize+0x49898 (C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x18004c47c)
#6 0x7ff9add4acb2 in GlobalAlloc+0x82 (C:\Windows\System32\KERNELBASE.dll+0x18005acb2)
#7 0x7ff9285d11c3 in DrvValidateVersion+0xaaa3 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_015fa42d67826549\nvoglv64.dll+0x180da11c3)
#8 0x7ff9289eae73 in vkGetInstanceProcAddr+0x4d653 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_015fa42d67826549\nvoglv64.dll+0x1811bae73)
#9 0x7ff75deeeb97 in DoRenderer::init C:\Users\phild\projects\gnometower\gnomes\do_renderer.cpp:704
#10 0x7ff75df37969 in DoRendererVParticle::init C:\Users\phild\projects\gnometower\gnomes\do_renderer_vparticle.cpp:345
#11 0x7ff75de6ece8 in GTApplication::run C:\Users\phild\projects\gnometower\gnomes\application.cpp:242
#12 0x7ff75de6f43b in main C:\Users\phild\projects\gnometower\gnomes\application.cpp:1012
#13 0x7ff75e091327 in __scrt_common_main_seh d:\a01_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
#14 0x7ff9af737033 in BaseThreadInitThunk+0x13 (C:\Windows\System32\KERNEL32.DLL+0x180017033)
#15 0x7ff9b00e2650 in RtlUserThreadStart+0x20 (C:\Windows\SYSTEM32\ntdll.dll+0x180052650)

Address 0x12a87a397f20 is a wild pointer.
SUMMARY: AddressSanitizer: bad-free (C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x18004d301) in _asan_wrap_GlobalSize+0x4a71d

Using the following setup (all updated to latest publicly available):
GeForce Game Ready Driver 511.23
Vulkan SDK 1.2.198.1
Win 10 19043.1503
Microsoft Visual Studio 16.11.9
Visual C++ 2019 - 00435-60000-00000-AA228

Is there anything I can do to get a response on this? Or at least verify it has the attention of the driver devs? I can’t use asan in my project until this gets fixed!

Happy to help and/or provide more information if needed!

Hello @phildo211 !

Thank you for so diligently reporting issue on the Vulkan drivers. I had a look at your past posts and I am sorry that you didn’t get the help you were looking for.

Could you maybe add a bit more context to help us understand what the actual problem is? It is not clear to me how the Asan run relates to the Vulkan driver. How would do you deduct from the above stack trace that the tool recognized the bad address handling within the NVIDIA driver?

It’s a “free unmalloced address” when I call vkGetInstanceProcAddr (see stack frame #8 in nvoglv64.dll). I guess it could be somewhere in ntdll.dll, but my assumption is it’s as shallow in the stack as is plausible, and it isn’t in my code (which just calls vkGetInstanceProcAddr and certainly doesn’t free anything).

This asan error comes up when I compile and run my project with asan, and (as I show above) points to an error through the nvoglv64.dll stack frame, and then the program terminates, so I am unable to use asan with my project until this is fixed.

Happy to provide any other information you’d like, but I’m not sure how to be more clear. Is there some assumption I’m making that is incorrect? Is nvoglv64 not indicative of the nvidia driver?

Thanks for the quick update.
To clarify, I will also bring this to the attention of the Vulkan driver team to get more input and to make sure we don’t miss any potential bug. Meanwhile let me try to help as much as I can.

nvoglv64.dll is the NVIDIA driver, correct. The reason I was asking was that there is an arbitrary memory address without exact signature of the function that tried to access that specific memory address. The stack trace ends in a kernel module, so it could happen anywhere in between.

Asan instruments DLLs that are not compiled with an Asan compiler flag (like nvogl or any of the Windows libraries) by checking for signatures of for example malloc, free, new or delete during runtime. For that to work together with your instrumented sources, the Asan runtime needs to be loaded before any part of the executable that was compiled with the -fsanitize=address flag. So please make sure you preload the Asan runtime. You should find information on how to do that in the MSVC Asan documentation.

Finally, the best way to help us investigate further is if you could provide a minimal reproducible example, meaning a small application that shows exactly the same behavior. That way we can hopefully reproduce it easily and find the potential bad free in our driver.

The reason why I ask all this is simply that we of course use all kinds of static and runtime analysis tools on our drivers. So I hope you can understand that I would be surprised if we overlooked a bad free in as essential a function as vkGetInstanceProcAddr.

Also to unblock you in the short term, if you would be able to use Clang for compilation, you can whitelist external libraries, see the CLang documentation on it.

Thanks for the helpful explanation!

At the time of the bug, I had simply enabled address sanitizer in visual studio’s project properties. It had worked for me in the past without any further modification.

I’m unfortunately not sure what you mean by “ensuring I preload the asan runtime”, and reading through the MSVC asan documentation I saw nothing regarding that.

However, I’ve since upgraded to VS 2022, and can no longer reproduce the bug (asan now runs with no error). I’m not sure what the issue was before, but I think it is reasonable to assume it was user error- I would love to better understand what steps I might have been missing, but for now my issue is resolved.

Thanks again for your responses- and I hope I didn’t waste too much of anyone’s time!

You most definitely did not waste anyone’s time, no worries.

My guess why it is working now would really be a mismatch in libraries needed by the compiler, driver and Asan. If you read through the documentation you will see, that Asan has a lot of both soft and hard requirements in terms of environment. That means that I doubt it was a user error but rather lack of proper setup help and documentation.

The preload comment comes from the final Note in one of the official announcement blog posts from Microsoft as well as some other comments I found. In the CLang/gcc world you have the LD_PRELOAD directive that allows you to specify the Asan runtime to be loaded before any other libraries. I am not certain if MSVC has something similar when building from the IDE, but at least you need to disable incremental linking, which is also mentioned in the Docs.

In any case, I am happy that you could resolve your issue! Best of success with your project!

Hi. I have the same problem after MSVC update to 14.29.30133. Looks like problem in address sanitizer library.
Builds with MSVC 14.28.29910 works fine.

Log:

==12084==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x11ca632b9e20 in thread T18
#0 0x7fff73dfd301 (C:\Qt\5.15.2\msvc2019_64\bin\clang_rt.asan_dynamic-x86_64.dll+0x18004d301)
#1 0x7ffff334f4e4 (C:\Windows\SYSTEM32\ntdll.dll+0x18001f4e4)
#2 0x7ffff3350b30 (C:\Windows\SYSTEM32\ntdll.dll+0x180020b30)
#3 0x7ffff335bed9 (C:\Windows\SYSTEM32\ntdll.dll+0x18002bed9)
#4 0x7ffff335b3c6 (C:\Windows\SYSTEM32\ntdll.dll+0x18002b3c6)
#5 0x7fff73dfc47c (C:\Qt\5.15.2\msvc2019_64\bin\clang_rt.asan_dynamic-x86_64.dll+0x18004c47c)
#6 0x7ffff0d4acb2 (C:\Windows\System32\KERNELBASE.dll+0x18005acb2)
#7 0x7fff54e22d4c (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180ce2d4c)
#8 0x7fff54fa3c25 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180e63c25)
#9 0x7fff54fa55bc (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180e655bc)
#10 0x7fff54b62e88 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180a22e88)
#11 0x7fff54b64163 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180a24163)
#12 0x7fff54b677ce (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180a277ce)
#13 0x7fff549f24ef (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x1808b24ef)
#14 0x7fff54ba5a9e (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180a65a9e)
#15 0x7fff54ba561c (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180a6561c)
#16 0x7fff54e24ed7 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180ce4ed7)
#17 0x7fff73e0c483 (C:\Qt\5.15.2\msvc2019_64\bin\clang_rt.asan_dynamic-x86_64.dll+0x18005c483)
#18 0x7ffff2227033 (C:\Windows\System32\KERNEL32.DLL+0x180017033)
#19 0x7ffff3382650 (C:\Windows\SYSTEM32\ntdll.dll+0x180052650)

Address 0x11ca632b9e20 is a wild pointer.
SUMMARY: AddressSanitizer: bad-free (C:\Qt\5.15.2\msvc2019_64\bin\clang_rt.asan_dynamic-x86_64.dll+0x18004d301)
Thread T18 created by T0 here:
#0 0x7fff73e0d278 (C:\Qt\5.15.2\msvc2019_64\bin\clang_rt.asan_dynamic-x86_64.dll+0x18005d278)
#1 0x7fff54e238ca (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180ce38ca)
#2 0x7fff54ba7554 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180a67554)
#3 0x7fff54f137e3 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180dd37e3)
#4 0x7fff54e17ab7 (C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_72959b4e763ac8f5\nvoglv64.dll+0x180cd7ab7)
#5 0x7fffdb860864 (C:\Windows\SYSTEM32\OPENGL32.dll+0x180020864)
#6 0x7fffdb860c9e (C:\Windows\SYSTEM32\OPENGL32.dll+0x180020c9e)
#7 0x7fffdb86331b (C:\Windows\SYSTEM32\OPENGL32.dll+0x18002331b)
#8 0x7fffdb87f4ef (C:\Windows\SYSTEM32\OPENGL32.dll+0x18003f4ef)
#9 0x7fffdb880510 (C:\Windows\SYSTEM32\OPENGL32.dll+0x180040510)
#10 0x7fffdb87f912 (C:\Windows\SYSTEM32\OPENGL32.dll+0x18003f912)
#11 0x7ffff0bd3a78 (C:\Windows\System32\gdi32full.dll+0x180063a78)
#12 0x7fff79dc4a6c (C:\Qt\5.15.2\msvc2019_64\plugins\platforms\qwindows.dll+0x180044a6c)
#13 0x7fff79dc33cb (C:\Qt\5.15.2\msvc2019_64\plugins\platforms\qwindows.dll+0x1800433cb)
#14 0x7fff79dc474b (C:\Qt\5.15.2\msvc2019_64\plugins\platforms\qwindows.dll+0x18004474b)
#15 0x7fff79d8de7f (C:\Qt\5.15.2\msvc2019_64\plugins\platforms\qwindows.dll+0x18000de7f)
#16 0x7fff79d8eb20 (C:\Qt\5.15.2\msvc2019_64\plugins\platforms\qwindows.dll+0x18000eb20)
#17 0x7fff79d8d3c2 (C:\Qt\5.15.2\msvc2019_64\plugins\platforms\qwindows.dll+0x18000d3c2)
#18 0x7fff70f8957f (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Gui.dll+0x18007957f)
#19 0x7fff716196ca (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x1800496ca)
#20 0x7fff71633c2f (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180063c2f)
#21 0x7fff71634a43 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180064a43)
#22 0x7fff71607e30 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180037e30)
#23 0x7fff715e498f (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x18001498f)
#24 0x7fff715e3a12 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180013a12)
#25 0x7fff70b12ac9 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Core.dll+0x1801d2ac9)
#26 0x7fff716124c9 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x1800424c9)
#27 0x7fff71619c69 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180049c69)
#28 0x7fff71618197 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180048197)
#29 0x7fff71617f3d (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180047f3d)
#30 0x7fff71619ab9 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180049ab9)
#31 0x7fff71619c83 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180049c83)
#32 0x7fff71618197 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180048197)
#33 0x7fff71617f3d (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180047f3d)
#34 0x7fff71619ab9 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180049ab9)
#35 0x7fff71619c83 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180049c83)
#36 0x7fff71618197 (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180048197)
#37 0x7fff71617f3d (C:\Qt\5.15.2\msvc2019_64\bin\Qt5Widgets.dll+0x180047f3d)
#38 0x7ff76b41b74f in main C:\workspace\enot\enot\apps\geostudio\main.cpp:42
#39 0x7ff76bc6bdb6 in WinMain (C:\workspace\enot\build-enot-Desktop_Qt_5_15_2_MSVC2019_64bit-RelWithDebInfo\apps\geostudio\geostudio.exe+0x14087bdb6)
#40 0x7ff76bc69a9d in __scrt_common_main_seh d:\a01_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
#41 0x7ffff2227033 (C:\Windows\System32\KERNEL32.DLL+0x180017033)
#42 0x7ffff3382650 (C:\Windows\SYSTEM32\ntdll.dll+0x180052650)

==12084==ABORTING

There is similliar problem discussed here:

I tried using setdll, it solved problem with Qt5Core.dll, but it not works with nvoglv64.dll.