See this Firefox bug report for details - investigation is still ongoing but multiple Firefox devs have suggested that this might be a bug in the Nvidia drivers:
[@ wlEglDestroyFormatSet] - Firefox 116.0.2 Crash Report - Report ID: fc09b7fc-8296-481b-85f4-6c0450230829
Martin, it looks like something is getting double-freed inside a wayland resize
To me this clearly looks like a bug in the Nvidia driver, a race condition.
If I read the stack right, the driver crashes because it handles updated linux dmabuf feedback tranches in the resize callback. The new tranch (without scanout tranch) is expected to get send when exiting fullscreen on Gnome, so that’s not a surprise.
The fact that it’s only observed so far withgfx.canvas.acceleratedenabled is likely due to timing differences, as reallocating/resizing buffers on the GPU takes time.
Just a guess, but the driver calls function
create_surface_context, indicating that it destroyed the old surface context - which may contained a reference to the format list to be destroyed inwlEglDestroyFormatSet, given that tranche data is surface related.
Not sure if any FF devs have reached out to the Nvidia Linux driver team about this yet, but as the person who filed and is experiencing this bug I thought I’d start a thread here in case other folks have been seeing it or have additional information.
I should mention that this continues to happen after upgrading from the 535.98 drivers to 535.104.05 drivers.