We are moving from vmware to citrix hypervisor 8.2 with nvidia T4 Cards (3 per Host). We have 6x Server 2019 RDSH VAD1912CU3 per Host. Each Worker with a T4-8A profile. We are happy with the benefits of the GPU for overall performance, except for system memory. We see a massive increase in Memory consumption in ctxgfx.exe and dwm.exe. Before our avg. ctxgfx.exe per User was ~70MB now with vgpu its ~265MB and avg. dwm.exe was 23MB and now is 253MB.
Are these numbers normal with vgpu? I opend a case with citrix but havent gotten any useful feedback yet.
Hi,
memory consumption depends on the protocol used. But at least for dwm.exe the increase is expected as the desktop window manager needs to do much more work. RDSH doesn’t support “native” NVENC, therefore the increase in ctxgfx.exe is also expected as you cannot offload to the GPU. It would be different for client OS like Win10 where you would see a decrease in ctxgfx.exe.
regards
Simon
Well that is kind of a bummer, i guess i buy more RAM then. In total our Sessions went up 800MB - 1 GB per User. In my old envoirment without vgpu i peak at about 25GB of commited memory for 10 Users and with vGPU its somewhere between 34 - 38 GB. That adds up …
How many monitors are the users running? Which resolution? In general, the numbers look really high. I just checked on my Citrix 2106 and Win2022 and the usersession consumes 57MB ctxgfx.exe and 17MB dwm.exe on 1 Full HD screen in idle with vGPU 13.
Which Citrix policies have you assigned for these users? Do you use bitmap codec (Thinwire) or Video Codec (H264/H265) ?
Those are the numbers, i was used to. Most users have 2x FullHD and we use most current vGPU 13.1.
I have to check Codec tommorow.
Thinwire with the option to use video codec for parts of the screen where Video or 3D Stuff is displayed.
OK, looks good so far. This would be exactly what I would recommend. ACR (actively changing regions) is most likely the best option in your case. Anyways, the numbers for sysmem consumption of the given processes seems to be way to high. By any chance can you test with the current 2109 on a VM to see if it might be related to the LTSR version ???
Another idea could be that there is an issue with ACR (consuming massive amount of resources). You should also disable ACR for a quick test to run Thinwire only and check again if this reduces the sysmem consumption.
Do you think i could just upgrade the VDA Agent on the Worker or would i have to build a complete Testenvoirment?
I havent been doing much on the protocoll and policy side of citrix (i mainly work on the Masterimages). How would i disable ACR via GPO. I couldnt find a value when i searched for it, maybe its because we use german templates?
Thank you so much for your valuable assistance!
Just the VDA is enough. I did further testing. Single screen with video playback (ACR) 100MB ctxgfx and 20MB dwm
Running 2xFullHD increases the processes to 130MB ctxgfx and 27MB dwm.
So it scales with 30% increase for the second screen. Removing the second screen goes directly back to the old numbers.
For better ACR policy understanding check my blog: [5 of 6] Mixed Codec (Adaptive Display) – GRID4ALL
I just talked to citrix about the issue, they asked me to check if consumption is as high when the vgpu is disabled. When vGPU is removed from the worker the memory consumption of DWM and ctxgfx is exactly as you described. (100MB and 20 - 30MB). Tommorow i will propably do three tests. First ACR policy, second new VDA Agent. Do you think downgrading nvidia vgpu software from 13.1 to 13.0 could be worth the effort? I dont really wanna mess with the citrix hypervisor to much as it seems to work super stable.
No need to downgrade vgpu software. I don’t expect NV to be the cause of the issue here.
Current state of the issue. Citrix collected info Hypervisor Team moved the ticket to VAD Team, VAD Team moved ticket after collecting more info back to Hypervisor Team.
Disabling ACR and trying different codecs did not change consumption noticeably.
Monday we try the VDA 2109 Agent.
Ok, thanks for the update. Why should this be related to hypervisor? I’m also running XS8.2 in my environment and I don’t see this behavior. I would assume this is related to the OS or the VDA. Another test might be to run the T4 in Passthrough to see if it makes any difference (rule out vGPU).
Just tested with 2109 results look pretty similar.
Hi, any chance to test with a vanilla OS (Win2019) to see if this is related to your image?
We will try with a vanilla system tommorow.
I made some screenshots showing the much higher consumption, maybe this helps?
Thanks but doesn’t really help. I even checked a Win2019 system to rule out the OS in my lab. 20-30MB dwm.exe usage.
Could you please also try a plain RDP session to see if at least the dwm stays in the expected range? I assume there is a dependency between ctxgfx.exe and dwm.exe whenever the issue occurs. Maybe a sort of memory leak or something like that. What I forgot to ask: Does this happen for specific sessions only? How much FB is in use when the issue occurs? Screenshot from GPUProfiler possible?
Thanks for all your effort.
dwm on rdp session (left Taskmanager, right Process Explorer) private bytes and working is still higher.
I will split this into 3 replies as there is a single Image limitation on new users.
Memory leak should occur over time, right? The high consumption starts right at the beginning of the session.
I would say that all sessions show this behavior (we also checked if there is a relation to workspace or endpoint type but the behavior seems identical).
In the screenshot we have 9 Users on the system (mix of single and double FHD) only Office/IT Staff some Office 2019, RDP, Webbrowsers, nothing fancy. Eventhough systen memory usage shows 55% the commit charge is at 100% and then the issue occurs.
We tried the same test but reduced the FB by 50% and hammered the system as hard as we could to rule out FB exhaustion causing the issue. Latency goes up obiviously but system is stable. Edit: less users to not run into the commit charge issue.
i build a vanilla server 2019 with vGPU Software, its doing windows updates right now. I will install VDA tomorow then test.
