Present logic of the Nvidia cards

I am wondering if the Nvidia cards send the previous frame to the Monitor when the underlying application fails to deliver the new frame until the Vsync point.

I think there’s three possibilities.

1-) GPU sends the previous frame to the Monitor if it did not received a new frame until the Vsync. This means GPU should create an identical buffer for the swapchain texture to copy into.
2-) GPU does not send the previous frame to the Monitor, the monitor itself contains the previous contents.
3-) GPU has two modes. Sends the previous contents IF the monitor does not hold the previous contents or GPU does not interfere and send the previous contents by itself

Thanks in advance for any directions

Hello @baris.uckardess and welcome to the NVIDIA developer forums.

Describing all the details would go a bit far for a forum answer, but I think you might be mixing up a few things.
The GPU does not “receive” a frame from an app, unless the app uses CPU rendering only. The GPU usually renders the next frame after an app is issuing render commands through some form of engine or lower-level API.

Whatever is rendered ends up in the GPU frame-buffer which the GPU will scan out to the Display according to the refresh rate of the Display. The Display does not store anything.

So if the GPU does not finish rendering a new frame in sync with the Display refresh rate, the old frame-buffer content will be displayed.

The swapchain is something that happens before scan-out. Usually systems run with double- or triple-buffered output. That means if new content is ready, the frame-buffer will be “swapped” with the next buffer in line and contain the new image.

There are a lot of docs, blogs and technical documents out there that will give a very detailed overview of the typical render pipeline of modern GPUs and Displays, I recommend reading through a few of them.

Thanks!

Hi Markus,

The Above statements of mine are oversimplifications since I am a new member of this community and I dont know the community here well enough.

In my case I am using Vulkan and waiting for graphics queue to finish or graphics device to be idle, so I can get a sense of when the submitted rendering commands are finished in the GPU. The problem is that even though I present well before the Vsync point(2-3 ms) it acts like I did not finish rendering into the frame buffer so gpu streams the previous frame buffer contents to the screen. The rate of this behaviour is pretty scarce in some setups and more aggressive in some others.

Here’s a test case. The test case uses 1080p resolution at 120-240hz. So we have our Vsync interval roughly 4.16ms-8.1ms, which is much less than 60hz.

Lets assume we have simple rendering application, it does not matter which API we using at this point.

Our application loop is simple.

-Set render target
-Clear rendertarget to red if it’s even frame index or clear rendertarget to blue if it’s odd frame index.
-Wait for gpu work to finish
-Present
-Increment frame index

Running this program in a perfect environment would result in constant violet color due to persistance of vision.

When the behaviour I have described above happens, the screen flashes with red&blue colors since the system is failed to stream the new contents of the framebuffer to the display so the old contents is streamed again and the previous frame buffer contents is displayed for 1 Vsync interval more.

I would like know if this behaviour can be avoided completely or not.

I do have source code for the test case above and I will be happy to share it.

Thanks for your time.

What monitor is connected? At 60Hz some people will still perceive distinct blue/red frames.

Is it possible that you are rendering too fast? Clearing the rendertarget does definitely not take 4-8ms. Try explicitly syncing the swapbuffer call to vsync and see what happens.

Feel free to share your test app, ideally in source, maybe that sheds some light on where something might go wrong.

I have attached the both source code and a premade executable for you to run it right away. The build instructions are in the ReadMe file in the source code. The test application I share with you uses Directx11 as it’s graphics API. I would recommend that you should use adapter index 0 at all times at the startup of the program where it accepts few inputs from the user such as which monitor you want to use for fullscreen rendering, the adapter index and the warmup time.

DX11PresentTest.zip (6.4 MB)

Here’s the monitors I have tested on so far. I use DP 1.3&1.4 and HDMI 2.1 for my test due to they support 240hz 1080p bandwidth.

  1. AOC C27G2Z 27"
  2. SAMSUNG 25" Odyssey G4
  3. MSI G27C4X 27"
  4. ASUS TUF Gaming 24”
  5. AOC 24G15N 24"
  6. Samsung 32" Odyssey G7

For your reply message,

Yes you are correct, at 60hz there will be constant flickering on the screen due to 60hz is not enough to create a healthy persistance of vision in this case. However as I have mentioned above I do not work on the refresh rates which are lower than 120hz due to they show distinct blue/red frames instead of a violet one.

In my test applications, I always sync with the Vsync whether it’s Directx11,Directx12 or Vulkan. It’s true that applications finish their rendering routines pretty fast since the only thing they do is to clear color the framebuffers.

Here’s few informations on the topic that I have accumulated in the lifetime of this project.

  • In windowed mode the flickers are more consistent and aggressive because of the Desktop Window Manager.
  • Flip modes do have an effect here. Present modes which use flip models instead of blit gives better results.
  • OS thread priorities seems to have an impact on the flicker rate.
  • Performance seems to play a role here. I have two setups where one has rtx4090 and the other has rtx2050. The setup with the rtx4090 has no flickers but the other setup has consistent flickers.
  • The DP or HDMI cables make difference. The one that shipped with the monitor performed worse than the one I bought externally( Paugge ENTMDP1415 VESA).

Not unless you run in exclusive fullscreen mode and keep focus by circumventing the WDM. I don’t know how to do that because you already use SetFullscreenState(), which should take care of that, but still it is only “windowed” fullscreen.

It should make a difference whether you are running with a single monitor setup or not. If Windows does not need to do any background window management and there are no other apps using GPU resources, you should get near perfect frame-times. I reproduced this locally by running on my normal 3 monitor setup vs closing all apps and using only one monitor. In the first case flickering comes very easily when moving the mouse.

Overall this is a very academic test and I do not really understand what you want to achieve with it. For game or application rendering your content is completely different. You either render slower than VSync and compensate by using tools like GSYNC, DLSS or Framegeneration. Or you render faster where you simply skip frames in your render-queue, but the delta between frames is not visible.

1) We wanted to omit the unnecessary details from you to not be confusing and focus on the frequent frame repeat issue which seems to be hardware-agnostic, but feel the need now to elaborate in order to not give the impression that this is purely for research purposes and does not have a specific real-life application:

The real code is for a custom SteamVR headset runtime. However, frame repeating seems to be apparent and frequent also with reglar 3d programs/games on regular 120-240Hz monitors connected to gaming laptops (these models tested: Asus G752VS, HP Victus 15, Aorus 15G-XC). So reducing or eliminating frame repeating serves real life purpose, it’s not for academic research.

We’ve also been discussing this with an open-source game engine developer (Panda3D), since the issue exists also on there, not just pure Vulcan/DirectX.

So we believe getting to the bottom of this will be beneficial to 3d and engine devs and is not some fringe use case.

Regarding the VR headset runtime, it does not render game frames, it takes frames provided by any SteamVR game and just post-processes them to be usable for a VR displays and optics. This includes reusing previous frames if the game failed to render new frames on time, or doing more advanced things like reprojection and lens distortion correction.

The test program provided is meant to illustrate, that even when nothing of the sorts is done and just a blank frame is presented each frame, there’s still an alarming number of repeated frames at the monitor (every 30 seconds or so) with these laptops even at 120Hz. Here’s the laptop logs, together with a spreadhseet of the results: Program result logs on 3 laptops, Dec 13 2023.7z - pCloud

Blue-red alternating colors is just for the test and is a good way to visually notice the issue clearly as it happens and in a video camera capture. They are easier to spot during testing than stutters, although stutters are still visible themselves.

Since your hardware seems too powerful to reproduce the issue, here’s some CGI videos we created to show the kind of stuttering that will happen with real life use cases. There’s videos made to play on a 120Hz, 144Hz, 180Hz and 240Hz monitors. The simulated frame repeat is exactly at the middle of the video.

frame-long repeat: normal monitor - pCloud

(view the refresh rate video matching your monitor’s, in fullscreen)

And here’s video camera captures of the test program on our devices. recordings Dec 8 2023 - pCloud (recorded with two cameras simultaneously, one 120fps and one 480fps, logs text files show the offending frame indexes)

Here’s few GIF animations we’ve generated from the videos, of just the few frames where the frame repeat happens. gifs of the artifacts (slow mo) - pCloud

To summarize what the test program does:

2 grids have a white card moving each frame, mouse cursor is also moved each frame, top grid card and the mouse cursor, unlike the bottom grid card, compensate for the repeated frame as soon as the program becomes aware that it has occurred.

When a frame repeat happens, you notice both the white cards and mouse cursors freezing in place, then:

  1. The mouse cursor jumps a position to compensate for the repeat.

  2. The top card moves a cell, only later jumps a cell to compensate. This is because the next frame was already presented and the 3d program couldn’t recall the next presented frame when it learned that the current presented frame had been repeated.

  3. The bottom white card just resumes as usual after the frame repeat, as it has not been programmed to compensate its position due to a frame repeat.

I’m sure you’ll agree that it’s not okay for the end user to experience this every 30 seconds or so on modern gaming laptops and at as low as 120Hz.

2) We are simultaneously testing the code using the VRWorks API, which is meant for VR and in theory should bypass DWM. We experience the issue with and without using VRWorks. Since VRWorks API needs an edit to the Windows registry to work with normal monitors, we are only supplying the pure DirectX version of the program to you for testing.

3) To clarify, the issue is not merely that only one frame sometimes stays longer / is displayed twice on the monitor than it should. There’s actually 2 mis-timed frames when this happens. The first frame gets repeated on the monitor, while the 2nd frame is already rendered and presented to the GPU and is displayed at the time slot of the 3rd expected frame (counted as 3rd if no repeating were to occur).

And often times, the frame is actually repeated twice or more in sequence, not just once.

4) SetFullscreenState does not seem to prevent the issue from happening. If there’s an alternative way to enable true fullscreen (I think you’re saying that what we do now is “only “windowed” fullscreen”?), please let us know.

5) Going one step ahead: is there any way to query the GPU or some other way determine a frame repeat is happening, before the repeating finishes? Is there any way to recall a presented frame or some other way halt the streaming of frames to the monitor when a frame is repeated once but hasn’t started streaming the next presented frame yet? This will at least solve the issue of frames repeated more than once and prevent later presented frames from being displayed later in time than they should have.

@MarkusHoHo

Hello Markus, I’m the manager for the project Baris is working on. Our company works on AR/VR optical hardware RnD and the project Baris is working on involves a custom SteamVR headset runtime for them. We are under NDA with Baris with regards to VRWorks API.

The hardware designs being developed involve pixel shifting and/or time-sequential optical foveation. In both cases, making sure the active optics inside the device are in sync with the PC program (PC VR runtime) frame index at the PC/GPU is crucial.

But as Baris explained, the described issue seems to be hardware and API-agnostic.
Right now Baris is able to compensate for frame repeats in the future frames, but it does not fix the visual artifacts that happen during the frame repeats.
We understand that the artifacts may not be completely preventable for 100% of the time, but we still hope it can be reduced greatly so that it doesn’t happen every 40 seconds or so on modern laptop hardware with no user background processes happening. As-is, we can’t really develop our pixel shifting and optical foveation hardware prototypes into viable products if the user is going to see a flicker or shift effect so frequently.

I just wanted to hop in and let you know that the code questions have a large hardware project behind them and they are not purely for learning or research purposes.
Me and Baris are available for providing any clarifications, data and additional measurements as needed.

Thanks

Hello @rghz1,

I appreciate you clarifying the context and I am certainly not putting this off as unimportant or purely academic anymore.

I do need to get some other people onto this case now because I am a bit at a loss,what might be going wrong here exactly. Either this is completely normal behavior for how the swapchain works or there is some flaw in the way we look at the render loop. The reason for that is simply that most PC games out there run (visually) smoothly without visible skipped frames. Animations, cut-scenes, demos, you name it, I have only seen effects like the ones you describe and show in applications that did some form of incorrect synchronization. But here to be honest I cannot recognize the possible issue.

Let me contact some people internally and see if we can get a fresh view on this.

Thanks!

Ok, I might have overlooked this part:

Can you list some app or game examples? As I stated in the beginning, frame repeats are normal if the Screen has no way of lowering the refresh rate and the game does not render as fast as the screen does scanout. There is no alternative than scan out the currently available framebuffer, even if that was presented before. The game will appear laggy and low fps, but how would you avoid that?

But I think your situation is different. I am just not clear yet how.

Thank you for the prompt response!

Even though we have spent many months on tests regarding this, it is very much possible that there may be some other way to code this to reduce (even if not eliminate) the frame repeat issue. We did have the issue happen more frequently previously until we tested all the possible modes to use with DirectX11 and then chose the one which provided the most stable results.

We also tried with existing game engines (Unity, Panda3D) and experienced the same, so whatever the case is, game engine devs will also find this info useful.

Sorry for not mentioning sooner, the issue seems to be present with GTX1070 (Asus Rog G752VS), RTX2050 (HP Victus 15) and RTX3070 (Aorus 15G-XC), but Baris seems to be unable to reproduce the issue on his RTX4090 systems (PC and laptop). Whether that’s because those GPUs are much more powerful and are able to handle 120-240Hz streams more reliably, or if it’s a solution due to how the new GPU architectures handle things, we don’t know.

Right now we’re hesitant to limit use of our hardware and software to RTX40xx series users, even if it was determined that these newer GPUs solve/greatly reduce the issue in general and not just the specific models we’ve tested on so far.

The issue seems to mainly come up at 120Hz and much more frequently at 240Hz so I’m not surprised it’s not reported or detected often. Still, solving this wouldn’t just be beneficial to VR users (120-180Hz) but also for competitive gaming use cases (120-240Hz).

Of course for VR it’s much more important due to repeated frames causing mismatch between the current shown VR view and the user’s real position/rotation, and because there’s no variable refresh rate in VR right now.

Even though what we do specifically (optical pixel shifting) is niche, it doesn’t seem like it’s only us trying to commercialize it: DigiLens Announces T-REx, Doubling Waveguide Resolution Within Existing Form Factor - DigiLens Inc.

Can you list some app or game examples?

I could try recording some games that support 120-240Hz rendering and rotate the in-game camera smoothly and then analyzing the camera captures frame by frame, but that will take some time and as you say, the games themselves may not be able to handle rendering at that rate, so it will be hard to isolate the stuttering due to the game/game engine and due to the runtime/GPU itself.

What I can tell now instead is that:

  1. We’ve created a Unity (DX11) and Panda3D (OpenGL) programs for displaying red and blue frames in sequence, and observed the same behavior.

  2. We’ve created a SteamVR runtime utilizing VRWorks API doing the same and tested it with regular 180-240Hz monitors and observed the same behavior.

  3. We’ve created a Unity SteamVR program displaying red and blue frames in sequence and ran it on an existing SteamVR headset (HP Reverb G2) with its own proprietary VR runtime and experienced the same behavior (of course less frequently, because it was 90Hz).

As I stated in the beginning, frame repeats are normal if the Screen has no way of lowering the refresh rate and the game does not render as fast as the screen does scanout. There is no alternative than scan out the currently available framebuffer, even if that was presented before. The game will appear laggy and low fps, but how would you avoid that?

But I think your situation is different. I am just not clear yet how.

The issue is not that the game / game engine is not able to provide frames on time. The frame repeat seems to be happening at the DirectX/Vulkan, GPU runtime or GPU stage which we don’t have access or control over. The APIs are telling our minimal test program to wait while our program itself is not busy and isn’t doing anything.
Same with the game engine tests, all they are doing is showing 2 differently colored quads, no real GPU resources are used up.

I hope this makes sense and sorry for the long message, I didn’t want to exclude probably needed info.

Hello @MarkusHoHo

We have done some more tests to potentially save you time on your own investigation:

Enabling Variable Refresh Rate (VRR) does not seem to solve the issue.

Our guess is it is due to one of two reasons (or both):

  1. The delay induced by the GPU runtime or firmware is much longer than what variable refresh rate monitors can support.
  2. There is a bug or limitation in the GPU/GPU runtime that preserves the issue even when VRR is enabled and supported.

We have tested on 2 laptops and 5 monitors that support either Freesync or Freesync Premium, so 10 total test setups. Each test was ran for 100 seconds.

The program code and executable, 480fps camera recordings, program logs and summary spreadhseet of the logs can be found here: Variable Refresh Rate tests Jan 17 2024 - pCloud

Laptops used:

  1. Aorus 15G XC-8US2430SH (2021)
  2. HP Victus 15 (2023)

Monitors used:

  1. AOC C27G2Z 27" - 240Hz - FreeSync Premium
  2. SAMSUNG 25" Odyssey G4 LS25BG402ENXGO - 240Hz - FreeSync Premium
  3. MSI G27C4X 27 - 240Hz - FreeSync Premium
  4. AOC 24G15N 24" - 180Hz - Freesync
  5. ASUS TUF Gaming 24” VG249Q1A - 165Hz - FreeSync Premium

(Amazon links to these monitors and laptops are available in the cloud folder file “Test hardware info.txt”)

This is what we did to enable variable refresh rate in code:

  1. Set swap effect to DXGI_SWAP_EFFECT_FLIP_SQUENTIAL
  2. Set the application to borderless window mode.
  3. Created and resized the swapchain with DXGI_SWAP_CHAIN_FLAG_ALLOW_TEARING flag.
  4. Passed 0 to vsync interval parameter field in the Present method.
  5. Passed DXGI_PRESENT_ALLOW_TEARING flag to flags parameter field in the Present method

So the issue does not seem to only affect VR use cases but also general 120Hz+ gaming, regardless of monitor used and whether they support Freesync/Freesync Premium or not.

Hi @ MarkusHoHo ,
I just wanted to let you know we haven’t found a solution here and exhausted the list of other places we could get professional consulting and help on this.