16 bit Overlays broken with X11 and 4XX series driver

I am running RHEL 7.3.
The Xorg looks clean. Option “Overlay” “True” is what I am using in the config. Composite and Xinerama are disabled. The only complaint I see is ACPI failed to connect to the event Daemon.
RandR is showing disabled (also I believe its version 1.4)
Shared Memory Pixmaps are disabled
There is also a message indicating NVIDIA(0): Enabling 16bit RGB Overlays.

libglxserver_nvidia loaded as well as libglx (mesa I think)

Ok, should be fine, then. Did you test if on the Pascal Quadro overlays work with the newer driver, i.e. whether this depends on hardware or driver version?

Just tried my P2000 and got the same results.

I just rolled back to 390.132 using the P2000 and the overlays work.
Assuming the 4XX series lineage maybe it broke at the branch to 410.57?

Hi nmcracker,
Please help to provide concrete repro steps so that I can try to recreate issue locally.

I’m not certain how best to answer this. I have a complex custom application that uses ffmpeg to handle my overlays.
Is there a good way to evaluate the system state using randr or xdpyinfo?
other than that a bare install of RHEL7.3 and the NVIDIA driver should be enough I think.

Will it be possible for you to share application or guide me with commands which can recreate issue.

sorry this is kinda difficult. What I understand is that we first search for a visual that has the overlay attribute then allocate the 16bit colormap for the visual. I would expect a small app that allocates the 16bit colors for the X drawable then draws a line should be able to reproduce the issue. I havent been able to get a test app shareable.

Between 390 and 410 the driver layout changed heavily, so seems some parts fell off.
Does glxspheres -o work as a repro scenario?

glxspheres -o fails to find a visual for both good and bad drivers.

please let me know if you have reproduced the issue or if I need to work harder at getting a tool build that won’t upset the powers that be.

Any updates I can pass along to my boss? I haven’t been able to get much traction beyond thinking that I am sending RGB555 to something that may be expecting RGB565.

If you can’t upload complete logs, can you please check xdpyinfo and verify that you have both 24-bit and 16-bit visuals showing up, and glxinfo and verify that there are some 16-bit GLX visuals there?

If you do see 16-bit visuals there, can you please test with an app other than your custom one? For example, atlantis -visual <vid> works pretty well, using the Atlantis app from the xscreensaver package.

The 16 bit visuals are present, xdpyinfo and glxinfo show 16bit 555 visuals.
Atlantis -visual 0x23 seems to work.

I have been trying to get a test application completed and now I am curious how Xv and putImage affect this. My test app as it stands now is doing what I expect but it is missing all the underlay components for video frame rendering which I think we do via Xv shared memory. There is a huge amount of code to skeletonize to try and get to a “failure” state so it may take a day or two. Is there anything special nvidia_drv does to handle Xv?

I finally reproduced the issue with a test application today. I don’t yet know if I can get permission to upload the source, What seems to be breaking the overlays is the pixmap I copy the overlay from.
something like this should do the trick.

XSetForeground(display, overlayGC, transparentColor);
XFillRectangle(display, overlayPixmap, overlayGC, 0,0, 100, 100);

XSetForeground(display, overlayGC, 0x7ffff); // white 555
XFillRectangle(display, overlayPixmap, overlayGC, 0,0, 74, 74);
XCopyArea(display, overlayPixmap, overlayWindow, overlayGC, 0,0, 100,100, 0,0);

Has there been any progress at reproducing this on your end?

Not yet, we will need application in order to attempt again or please help us to guide to any open source application which recreate same issu.

here is the zip of my test app.

nvidiaZip.c (3.8 KB)

Hate to keep being a bother but did my test app help?