16 bit Overlays broken with X11 and 4XX series driver

I just received my new Quadro RTX 4000 for my Linux box and found that my Video Overlays are not working anymore. My X server is running 1.19.3 and I have tried rolling back the drivers all the way to 410.78. This was previously working with my Quadro P4000 and the 384.59 driver.

Whats interesting to note is that I am getting the 16bit Color map from the X Server but when they render they look almost like the server is expecting 24bit color. white objects are coming out aqua.

My Xorg.conf hasn’t changed much between cards and I don’t see any extra documentation to indicate a change to my config is necessary.

Does anyone have any thoughts as to what I may have incorrectly setup or did the drivers break this feature?

Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post. You will have to rename the file ending to something else since the forum software doesn’t accept .gz files (nifty!).

Unfortunately due to restrictions at work I can’t upload the report.
is there anything specific I can look at and hopefully type by hand?

The Xorg logs should be checked if the overlay option is correctly applied and the driver doesn’t complain about anything. Which distro are you running?

I am running RHEL 7.3.
The Xorg looks clean. Option “Overlay” “True” is what I am using in the config. Composite and Xinerama are disabled. The only complaint I see is ACPI failed to connect to the event Daemon.
RandR is showing disabled (also I believe its version 1.4)
Shared Memory Pixmaps are disabled
There is also a message indicating NVIDIA(0): Enabling 16bit RGB Overlays.

libglxserver_nvidia loaded as well as libglx (mesa I think)

Ok, should be fine, then. Did you test if on the Pascal Quadro overlays work with the newer driver, i.e. whether this depends on hardware or driver version?

Just tried my P2000 and got the same results.

I just rolled back to 390.132 using the P2000 and the overlays work.
Assuming the 4XX series lineage maybe it broke at the branch to 410.57?

Hi nmcracker,
Please help to provide concrete repro steps so that I can try to recreate issue locally.

I’m not certain how best to answer this. I have a complex custom application that uses ffmpeg to handle my overlays.
Is there a good way to evaluate the system state using randr or xdpyinfo?
other than that a bare install of RHEL7.3 and the NVIDIA driver should be enough I think.

Will it be possible for you to share application or guide me with commands which can recreate issue.

sorry this is kinda difficult. What I understand is that we first search for a visual that has the overlay attribute then allocate the 16bit colormap for the visual. I would expect a small app that allocates the 16bit colors for the X drawable then draws a line should be able to reproduce the issue. I havent been able to get a test app shareable.

Between 390 and 410 the driver layout changed heavily, so seems some parts fell off.
Does glxspheres -o work as a repro scenario?

glxspheres -o fails to find a visual for both good and bad drivers.

please let me know if you have reproduced the issue or if I need to work harder at getting a tool build that won’t upset the powers that be.

Any updates I can pass along to my boss? I haven’t been able to get much traction beyond thinking that I am sending RGB555 to something that may be expecting RGB565.

If you can’t upload complete logs, can you please check xdpyinfo and verify that you have both 24-bit and 16-bit visuals showing up, and glxinfo and verify that there are some 16-bit GLX visuals there?

If you do see 16-bit visuals there, can you please test with an app other than your custom one? For example, atlantis -visual <vid> works pretty well, using the Atlantis app from the xscreensaver package.

The 16 bit visuals are present, xdpyinfo and glxinfo show 16bit 555 visuals.
Atlantis -visual 0x23 seems to work.

I have been trying to get a test application completed and now I am curious how Xv and putImage affect this. My test app as it stands now is doing what I expect but it is missing all the underlay components for video frame rendering which I think we do via Xv shared memory. There is a huge amount of code to skeletonize to try and get to a “failure” state so it may take a day or two. Is there anything special nvidia_drv does to handle Xv?

I finally reproduced the issue with a test application today. I don’t yet know if I can get permission to upload the source, What seems to be breaking the overlays is the pixmap I copy the overlay from.
something like this should do the trick.

XSetForeground(display, overlayGC, transparentColor);
XFillRectangle(display, overlayPixmap, overlayGC, 0,0, 100, 100);

XSetForeground(display, overlayGC, 0x7ffff); // white 555
XFillRectangle(display, overlayPixmap, overlayGC, 0,0, 74, 74);
XCopyArea(display, overlayPixmap, overlayWindow, overlayGC, 0,0, 100,100, 0,0);