Regression: crash during OpenGL shader compilation on 525.60.11

Graphics / Linux Linux
1

Hello,

there seems to be a crash in 525.60.11 during shader compilation. This has not been an issue before. The last known working driver version is 510.108.03.
The issue was originally reported on 525.60.11 by a user using the Manjaro package, but I confirmed it on Ubuntu 22.04 using the Nvidia binary installer.

Attached are the offending shaders and the nvidia-bug-report output.
opengl_shader.zip (38.1 KB)
nvidia-bug-report.log.gz (404.0 KB)

Here is a backtrace of our application crashing due to the issue:

#0  0x00007fffbd93cd08 in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#1  0x00007fffbd957b71 in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#2  0x00007fffbd91ae0e in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#3  0x00007fffbd958f97 in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#4  0x00007fffbd8cdabe in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#5  0x00007fffbd8ce39e in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#6  0x00007fffbd8d399e in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#7  0x00007fffbd8d4395 in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#8  0x00007fffbe82cff5 in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#9  0x00007fffbe8302db in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#10 0x00007fffbe8346bf in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#11 0x00007fffbe80cc23 in  () at /lib/x86_64-linux-gnu/libnvidia-glcore.so.525.60.11
#12 0x00007fffee992646 in gl::linkProgram(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (glprogram=147, message="")
    at /home/karol/overte/libraries/gl/src/gl/GLShaders.cpp:410
#13 0x00007ffff2310dd6 in gpu::gl::GLBackend::compileBackendProgram(gpu::Shader const&, std::function<bool (gpu::Shader const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gpu::Shader::CompilationLog&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)> const&) (this=0x555559ff1ad0, program=..., handler=...)
    at /home/karol/overte/libraries/gpu-gl-common/src/gpu/gl/GLBackendShader.cpp:149
#14 0x00007ffff2326745 in gpu::gl::GLShader::sync(gpu::gl::GLBackend&, gpu::Shader const&, std::function<bool (gpu::Shader const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gpu::Shader::CompilationLog&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)> const&) (backend=..., shader=..., handler=...)
    at /home/karol/overte/libraries/gpu-gl-common/src/gpu/gl/GLShader.cpp:43
#15 0x00007ffff22fa6c1 in gpu::gl::GLBackend::syncProgram(std::shared_ptr<gpu::Shader> const&) (this=0x555559ff1ad0, program=std::shared_ptr<gpu::Shader> (use count 3, weak count 1) = {...})
    at /home/karol/overte/libraries/gpu-gl-common/src/gpu/gl/GLBackend.cpp:1012
#16 0x00007ffff2055286 in gpu::Context::processProgramsToSync() (this=0x555556e9d9a0) at /home/karol/overte/libraries/gpu/src/gpu/Context.cpp:382
#17 0x00007ffff7d5c5eb in operator()() const (__closure=0x7fff057f8ea0) at /home/karol/overte/libraries/display-plugins/src/display-plugins/OpenGLDisplayPlugin.cpp:599
#18 0x00007ffff7d5f99e in OpenGLDisplayPlugin::withPresentThreadLock<OpenGLDisplayPlugin::updateFrameData()::<lambda()> >(struct {...}) const (this=0x555557675740, f=...)
    at /home/karol/overte/libraries/display-plugins/src/display-plugins/OpenGLDisplayPlugin.h:185
#19 0x00007ffff7d5c735 in OpenGLDisplayPlugin::updateFrameData() (this=0x555557675740) at /home/karol/overte/libraries/display-plugins/src/display-plugins/OpenGLDisplayPlugin.cpp:590
#20 0x00007ffff7d5d758 in OpenGLDisplayPlugin::present(std::shared_ptr<RefreshRateController> const&)
    (this=0x555557675740, refreshRateController=std::shared_ptr<RefreshRateController> (use count 1, weak count 0) = {...})
    at /home/karol/overte/libraries/display-plugins/src/display-plugins/OpenGLDisplayPlugin.cpp:698
#21 0x00007ffff7d66bf2 in PresentThread::run() (this=0x55555bab0450) at /home/karol/overte/libraries/display-plugins/src/display-plugins/OpenGLDisplayPlugin.cpp:198
#22 0x00007ffff33b5ca1 in QThreadPrivate::start(void*) (arg=0x55555bab0450) at thread/qthread_unix.cpp:329
#23 0x00007fffee134b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#24 0x00007fffee1c6a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
2

Please help to share repro steps which will help to duplicate issue locally followed by debugging.

3

The easiest repro I know is to get our AppImage from https://github.com/overte-org/overte/releases/download/v2022.09.1/Overte-2022.09.1-x86_64.AppImage and just run it on 525.60.11.
It will crash during shader compilation.
The same issue happens with a fresh debug build, so it isn’t an issue with AppImages.

4

Thanks @juliangr for sharing repro application.
I can reproduce issue locally with below error message -
[752341.363068] Presentation Th[3348844]: segfault at 2a0000002c ip 00007fa2115ffe58 sp 00007fa131def440 error 4 in libnvidia-glcore.so.530.00[7fa211523000+281c000]

I have also filed a bug 3916982 internally for tracking purpose.

5

We have root caused the issue.
Fix is incorporated in future release driver.