Looking for advice on optimal config for latest-gen Citrix Xenapp vGPU solution

Virtual GPU Forums More vGPU Forums XenApp
21

Hi

A XenApp / RDSH workload is classed as having lower-end throughput, so using an 8x PCIe slot is absolutely fine. Sorry if I wasn’t being clear, but you can use both x8 and x16 PCIe Slots for this type of workload with no notable difference in performance.

Page 71 of this PDF: https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00008180enw

If you’re looking to use other GPUs in the future that are currently unknown, then they may require a different Riser configuration to get the ideal density / performance.

Just out of interest, what is the use case for the PCIe SSD, rather than using the SAS / SATA SSD drive bays at the front of the chassis?

Regards

MG

22

OMG the fact that x8 slots are supported is complete news to me. Where on the T4 pages is this mentioned ?? Nowhere have I seen this mentioned. I wish I had known that up front. Nevertheless it’s not a big problem as it would ‘only’ require replacing the risers while we still have the best possible servers in terms of density scaling and it’s good to know that if no better cards (with more mem) come out, we can still replace risers and go that route if needed.

So thank you for that useful information. Better late than never ;) At this point I basically designed by scaling out a bit broader by having more HP Server which circumvents the problem but in case we run into a bottleneck there i’ll keep in mind that I still have a way out this way without having to add more servers.

the PCIE SSD’s reach performances in a completely different class on their own because of the PCIE interface, uncomparable to SAS/SATA. At this point they’re in experimental phase for some applications.

Either way I definitely would rather not sacrifice the slot unless I really need to but it’s good to know that it’s possible to change the risers and go up to 7 PCIE 8x slots.

23

Hi

No problem, glad it’s useful info and opens up more possibilities for you :-) The usage of x8 vs x16 really is workload dependant, but in this case you’ll be fine to use them both.

With XenApp / RDSH workloads, you’re unlikely to generate a high enough IO or require response times quick enough over 12gbs SAS SSD to justify the additional expense and PCIe Slot sacrifice that NVMe brings. You should have about 24x SAS slots available on the front of the Server (depending on configuration), for this workload my recommendation would be to use those whenever possible keeping the PCIe Slots free for additional GPUs. By all means experiment with NVMe, but remember they will definitely require x16 PCIe Slots to get the best out of them, so may not be very helpful when you want to add density with GPUs. And yes, they’re pretty expensive too!

If you were doing something fancy with Mellanox like RDMA and RoCE, or you wanted to use an NVMe drive as a vSAN Cache disk, then that would be worthwhile, but for XenApp / RDSH 12gbs SAS SSD is the best "bang for buck".

Now … Go get those Risers changed and get some more GPUs installed! ;-)

Regards

MG

24

hahaha…compelling argumentation mate ! =P

25

Thank you for all this great information, i’v been reading through it all.
This just show how "underpowered" our setup is now.

Profundifo, I would be very careful to choose a too small profile for your setup. This could hurt you a lot in a few years, like it has done to us now.

In 2017, we came to this solution:

Citrix XenApp VMs, each with this config:

7 vCPUs (2.4GHz)
60GB RAM
Nvidia M6 (1A profile, 1GB framebuffer)
Users: 10

This setup ran like a dream from 2017-2019. We used 85-95% framebuffer all the time, but it was running really good. The GPU processing power was offloading the CPU a lot, we could play full screen videos etc…

Then in September 2019, we had to update the server firmware, to stay on supported versions. That made the snowball roll on everything else needed to be updated as well. After we updated and everything was within supported versions of each other, the Citrix VMs started to become unreasonable slow as soon as more than 4-5 user was logged on (and would eventually crash, as a few more users was logged on).

The past year we have tried every possible combination of Citrix VDAs/GRID drivers/XenServer etc…, it’s not working for us. We have been forced to disable ALL of our GPU to have a running production environment.

Both Citrix support and Nvidia support have now given up on us, they can’t help us. We simply don’t have enough framebuffer to support the workload anymore.

So, for a year now, our expensive Nvidia M6 cards have been collecting dust…

26

Hi

Citrix "XenApp" (Not XenDesktop?) with a 1GB Profile? I hope that’s a typo, because that’s a crazy thing to do.

All XenApp / RDSH VMs using the M6 in a production deployment should be using the full 8GB (8A). Nothing less than that, unless it’s in a Dev environment for testing.

I’m surprised you managed to get more than 1 user on it :-)

Regards

MG

27

Hi MrGrid

I really do mean Citrix XenApp, on a multi session OS (Server 2016). We have about 8-12 users on each of those XenApp VMs.

At the time (2017), when this was configured, no one even questioned that 1GB frame-buffer was enough for 10 users. We primarily choose to buy the GPU’s to offload our CPU resources and we had great results.

My colleague have done a presentation at NVIDIA GTC on our setup (at NVIDIA’s invitation). Nvidia GTC being the GPU Technology Conference… Nvidia also did a "Nvidia customer spotlight" video on our solution and Case study… Would be stupid of Nvidia to do, if this was not a valid setup.

28

Hi

I would have questioned it :-) In fact, I would have strongly advised you not to do it.

I assume you’ve tried using the 8A profile instead of the 1A now?

Regards

MG

29

If we had been told that we should use even the 2A profile instead, we would not have bought the GPUs. The cost would have been way to much.

We have tried all the A profiles, only 4A and 8A are usable now.

Using the 8A profile, would reduce the amount of VMs from 152 to 19.
Which would reduce our (vGPU enabled)capacity from 1520 users down to 190.

Either way, we are now forced into choosing:

  • Disable all GPUs and get 0 value from the GPUs.
  • Create a new delivery group, limiting 190 users with access to GPU.

This is extremely disappointing.

30

Hi

Can you please give me some additional details about the configuration and versions currently in use.

Which version of XenApp: ?
Which XenApp (Windows) Operating System: ?
Which version of XenServer: ?
Which vGPU version: ?
Which Tools are you using to monitor the resources capacity and performance: ?
What type of Applications are being used: ?
Physical server hardware used and generation: ?

Regards

MG

31

Sure, I’ll give your our current setup, but know that we have tried a large number of combinations.

Which version of XenApp: 1906.2

Which XenApp (Windows) Operating System: Server 2016, version 1607 build 14393.3866

Which version of XenServer: CH 8.2 update 1

Which vGPU version: Grid 11 (and corresponding windows driver)

Which Tools are you using to monitor: Depending on what we are looking for: Nvidia-smi (xenserver and windows), ControlUp, GPUProfiler, Remote display Analyzer, Resource Monitor… etc

What type of Applications are being used: Internet Explorer, Chrome, Microsoft Office suite and various applications not utilizing GPU. A few have applications like AutoCAD and Adobe CC, but we have excluded them from our tests.

Physical server hardware used and generation: Cisco B200M4S (Xeon E5-2680v4, 512GB ram, Nvidia M6). Storage is handled by NetApp AFF-A300 (SSD network storage).

I hope that Profundido got his answers, since I’m hijacking his thread. :)

32

Thanks for the details. Initially, I can’t see anything obvious. XenApp and W2016 are a bit old compared to current versions of W2019 and CVAD 2006, but they are still supported. vGPU and XenServer versions are obviously fine.

Personally, with Citrix deployments I always run CR as well, and never LTSB. I always keep my Citrix environments completely up to date. But as said, what you’re running is supported.

If memory serves me correctly, the B200 M4 only supports a single M6 (M4 architecture is getting on a bit now), this would be quite a limitation, especially as the B200 M5 supports 2x GPUs and not only that, but they’re twice the amount of framebuffer. That’s a big jump forwards.

You mention:

Are those Apps not using the GPU because you’ve configured them that way, or do you want them to use the GPU but they’re not?

Silly question, but I take it you’ve enabled the use of a GPU through a Group Policy? (The answer should be Yes).

When you allocate the 8A vGPU Profile, how many users do you get on 1 XenApp VM and what kind of Apps are they running to generate that load? I’m assuming you hit the GPU Framebuffer limitation before any other component? You can use Director to pull basic vGPU metrics, that will be sufficient for this as all you’re doing is measuring capacity.

Regards

MG

33

We do plan on trying WS2019 in the near future, since WS2016 1607 is getting quite old.
It just requires a bit of planning and testing.

We have tried CVAD 2003, made no difference. I don’t think we have tried CVAD 2006 yet.

Yes, the B200M4 only supports a single M6.

Those programs are just not utilizing GPU. We have over 200 applications in total, I’m not going to list them all. :)

Yes, we have enabled that policy. We have also been suggested to try and use different drivers(Display adapter priority and monitor creation in Citrix sessions), but to my understanding that’s not available for XenApp. I was only able to change this in our XenDesktop environment.

When we use the 8A profile, we have the same load evaluation for that VM as all others. That leaves about 7-10 users on each VM. The load evaluation is based on: total user sessions, CPU usage and memory usage.

We did try and have 15-20 users on a 8A profile, but that just completely maxed out the CPU. Which lead to another set of problems. For applications it’s nothing special, we focus on just getting browsers and office applications to work. Then we can try out more demanding applications.

I’m pretty sure it’s an issue with frame buffer exhaustion and a extreme performance degradation when video memory is "swapping" to system memory.

34

Hey guys,

just a quick update: the POC I ran in this thread was based on our then existing 2012R2 golden image and resulted in up to 20ish users on a 8A profile. However, when going productional now using our brand new 2019 golden image I have noticed that the framebuffer requirements have basically doubled.

We are succesfully running up to a maximum of 20users on 16A profiles. So I basically ended up dedicating 1 full T4 card to each virtual Xenapp VM. Those VM’s currently run with 80GB mem and 16cpu cores. They handle everything I’ve thrown at them beautifully so far. At 20 concurrent users I’m at 90-95% framebuffer usage. I then have 8 of those VM’s per physical host server.

I’m targeting and sizing for 15 users per VM on average, up to a hard maximum of 20, set by policy.

Been running for weeks with 18-20 users (initial stress test) on 1 server to validate and it seems rock solid.

35

Hey Profundido

Thanks for the update! Glad to hear you bought some more PCIe Risers … ;-)

Really happy that it’s worked out! That’s great news! Sounds like your testing and capacity planning is working well! Great job!

Yes, W2012 R2 to W2019 is a big jump in terms of OS and resource requirements. This is the kind of thing that we need to account for when building a new platform. Don’t build it for todays OS and user density, try and build it for the next 3 - 4 years, which means factoring in things like Memory, CPU Core capacity, GPU Framebuffer, Storage I/O etc etc. If that’s not possible, make sure you can add additional resource by scaling out, instead of up.

Hi Athomsen

Unless it’s changed with recent versions, the Citrix load evaluators don’t take GPU utilisation into account, and as this is your limiting factor, you may want to remove them from a GPU enabled XenApp VM and re-asses to make sure your density is correct.

Typically, for a VM with an 8A vGPU, I start with 8 vCPUs and 32GB RAM. That’s a decent base specification to work from as a starting point. If you were starting with a larger GPU then you’d use a few more CPU Cores and bit more RAM to add balance. Then install the Apps, optimize the OS (Citrix / VMware Optimizer are both pretty good) and start loading up gradually to test with. Scale CPU and RAM resources up or down until you max out the GPU. As the Framebuffer is the only element that cannot be increased (in this situation), you should build your entire specification and user density around that single element. Max out the Framebuffer, then reduce density till it’s at about 85 - 90% to leave a little head room, then look at what your CPU and Memory are doing and tweak accordingly to find the balance. Once you have that, you can then work out how many VMs / Servers you’re going to need. Just remember that 8GB is not a lot of framebuffer today, so your density will be lower, and if you’re planning to move to W2019, then that will more than likely require additional hardware, although the jump from W2016 to W2019 may not be as severe as going from W2012 R2.

Also, the point of running CR vs LTSB is that you use the most recent version, if you’re not doing that then you may as well run LTSB (which I would not advise because you’re running vGPU). Luckily Citrix makes it pretty easy to keep things up to date. License Servers, Storefront, Director and Controllers are an in-place upgrade. The VDA can be a bit more of a pain depending how many Catalogs you have, but I typically uninstall / reinstall the VDA rather than try to upgrade it. I also include the vGPU drivers in that process as well.

Let us know how you get on!

Regards

MG

36

Hey Profundido

It’s beginning to sound close to 1GB framebuffer for each user now.
On my current setup, that would mean 1-2 users on each VM (1A profile).

I would need to scale out with 10 times more hosts, to handle the same load we had a year ago.

Hey MrGrid

Yes, the limiting factor is definitely the amount of framebuffer. I don’t see how we solve that, without spending a ton of money on new hardware. For now, we have to keep things running with the hardware we have. In a year or two, we will have a budget for new hardware. I don’t see Nvidia GPU’s being a part of our setup in the future.

We do have a pretty neat setup for building our Citrix image. We can build it completely from scratch in a matter of hours, as everything is scripted. We never have to deal with issues with upgrading/uninstalling software.

37

hey Athomsen,

I’m sorry to hear you basically ran into a hard dead end with your existing design. Thinking that last year when I was doing the POC I was still considering older gen Nvidia cards but since I personally know from over 20 years experience in architecture design just how fast a working setup can become outgrown or obsolete I have gotten into the habbit of thinking lightyears ahead and always shooting my arrow for the highest most flexible design that leaves me the most possible options to adapt and handle.

From what I have personally observed in terms of load and behaviour but also codecs support (check the Nvidia support matrix) for instance I’m even surprised that Nvidia would still even sell anything less than T4 cards. And even then I stick to my earlier conclusion that even that is ‘prone to fast obsoletion’ solution anno 2020 simply because we cannot scale out the total framebuffer memory adequately with the current solution (T4 cards with 16GB mem)

Bottomline Nvidia as a corporation is imho still missing the point that they don’t have a solid scalable solution for the RDS/Citrix market which is too big to ignore anyway.

Either we need something like a T4 but with at least 32GB or more mem or an Ampere-based successor that does just that.

And then there’s the point of the cost. Since current T4 gen cards are so limited in memory, forcing customers to stack up to 7 per physical host server I believe their cost should come down greatly because this is just too expensive for the use case. The 7 cards per server alone can easily cost more than an entire Citrix/RDS server itself. Budget managers tend to see this ‘extra feature’ way overpriced atm.

The tech works though, and it’s worth it if you can get it. In your case I would wait a bit until an Ampere-based solution with more mem comes out, see if that fits your future design and ask for a potential tradeup program if there is such a thing.

38

Thanks Profundido,

Good feedback. The Framebuffer limitation has been flagged many times. I do not know if there will be an Ampere upgrade for the T4 (I hope so as there have been many requests), but we’ll have to see. GTC is coming in October, maybe something will be announced then. I notice that there has only been 1 Tesla (A100) and no Quadros revealed yet, so they are coming at some point …

Hi Athomsen

I know you mention it works, but honestly, 1A Profiles are really not the way to go with this. It’s the wrong configuration for XenApp deployments. 8A is the Profile you should be using with this GPU for XenApp, then design around it regarding CPU and RAM as mentioned above. Again, I know you mention you can’t do that at this stage, but I’m just letting you know that this is how it should have been from the start.

It sounds like there are a few factors that are at play here. Unfortunately, whoever it was that set up the environment originally / carried out the original load testing / gave the original advice, has set you on the wrong path to lead you to this point where you can’t scale up or out. You mention no one flagged 1GB Profiles as being a potential problem with XenApp, and I can absolutely assure you that someone should have.

The M4 architecture (B200 / C240 / C460) was Ciscos’s Gen 1 for supporting meaningful GPUs (Maxwell) so they were / are limited. Having refreshed my memory now (I work with a lot of different OEMs), the B200 only supports 1x 8GB GPU. Meaning that if you wanted to scale past that, you can’t scale up so you have to scale out by purchasing additional B200 M4’s. This falls into the planning I mentioned above about leaving headroom in the config (very difficult with an 8GB GPU) or having the ability to scale out and why I mention about the original testing, capacity planning and guidance received. The problem with Blades is that they are extremely limited in their configuration options, whereas with standard rack servers you have the choice of multiple components. So when choosing Blades, it’s really important to ensure you have sufficient headroom to scale up or in this case out as the environment matures, and if that’s not possible, then standard rack servers may be a better option to allow the flexibility. If you were to move to the B200 M5 (or M6 when it eventually arrives at some point), then this would allow you to use at least 2x 16GB GPUs per Blade which would obviously help and as you can see from Profundido, he’s having to use an entire 16GB GPU for 15 - 20 users to support his specific applications / user density requirements, however he’s also having to use a very different configuration and set of technologies to obtain those numbers. But each deployment has it’s own specific requirements, so they can only be compared at face value.

Regarding not using a GPU in your next deployment - speaking as a Consultant, Engineer, Architect (my previous roles) and outside of my relationship with NVIDIA - that would be a big mistake. The GPUs bring more to the environment than just Framebuffer and 3D Graphics, they also bring dedicated hardware encoding and protocol choice and optimisation (you’ll know that already with your Citrix experience). And as Operating Systems develop with every update and require more processing resources, and as remote working continues to increase and become the norm, not having at least dedicated encoding will cause problems. And just a quick point on the encoding … GPUs obviously have multiple components on them depending on their given workload. Simply adding more Framebuffer is not always the answer, as the more users that go on the GPU, the more sessions need to be encoded placing additional load on the encoders, so these all need to be considered as we try and balance the components. The goal would be to have everything reach capacity at the same time, however as I’m sure you can appreciate, this is extremely difficult to do with such a variety of workloads and use cases. As mentioned above, hopefully new additions will come fairly soon, as there is a lot of request / demand for them.

My advice, put this platform down to experience and try and get the best out of it until you upgrade. Understand the issues and the decisions made when building this platform. Use this experience when building your next platform to make sure your next one has headroom in each profile and the ability to scale up or out. I can guarantee that your next platform will be superior to this in every way. Not because of the newer technologies, but because of your understanding of the limitations of each component.

Regarding your current platform, are you able to see which specific Apps or processes are consuming the Framebuffer so we can understand more about the utilisation?

Happy to have a chat with you off line about this, drop me a DM if this is of interest. I’m based in the UK, so our time zones are only 1hr apart.

Regards

MG

39

The cost for using A8 profiles instead of A1, back in 2017, would not even have been considered an option for us. It would have been ridiculously expensive.

Yes, if Nvidia would design a GPU with 64GB memory on a low end GPU (to keep cost down), that would be awesome. For us, a GPU like the M6 with 64GB memory, would be perfect. We don’t even come close to utilizing all the processing power of that GPU, but the memory is insufficient.

Have you thought about how you would handle the frame buffer requirement doubling in a year or two?
We don’t really know what will happen at this point. But for us, the requirement went up by 800% in less than 2 years.

40

Hi

Yes, I always consider the lifecycle of the platform. With my role in particular, it’s critical and I have no excuses for getting it wrong. I’ve been working with vGPU technology since its inception, and I constantly advise and remind OEMs, Partners, Customers and other Professionals who are less experienced with vGPU than myself, if they under-size the Framebuffer incorrectly, even by just 1 Profile size, they halve the density of the entire platform, let-alone if they need a bigger profile jump than that. Framebuffer planning is absolutely critical, because unlike CPU and RAM where additional resources can be added, Framebuffer is a hard limit (the GPU has a finite amount of it) that cannot be altered which is why we build the other specifications and capacity planning around it when using smaller GPUs.

With this in mind, I always try to plan for OS upgrades, Application upgrades, existing Monitor count and future 1080p - 4K upgrades, VDI / Hypervisor upgrades, the type of end-points and allow additional user capacity within a single vGPU Profile if possible. I then make sure I can either scale up (by increasing CPU / RAM or physical GPUs into a single server at a later date, if I haven’t already accounted for it in the original specification), or if that’s not possible, make sure the environment can scale out by purchasing additional servers. At the same time, if you are sure that over the lifecycle of the platform that you aren’t going to move from W2016 > W2019, you aren’t going to change Hypervisor or VDI stack because you’re using LTSB, and you know roughly what your user capacity will be + / - 10-20%, then these are much easier parameters to design with, but you face obvious limitations if you deviate from that later on in the lifecycle.

In my opinion, from the information you’ve kindly provided so far, the reason that it would have been expensive in 2017 for you, was because the decision was made to use B200s and the M6 rather than C240s and the M10. Meaning that you have to purchase an entire Blade to scale out just 1 GPU, because it’s not possible to scale up. That said, I obviously do not know your infrastructure, the decision for using Blades in the first place or what other workloads the Blades are supporting, I’m assuming they must be doing other tasks as they have 512GB RAM in them. It’s these kind of decisions that are important to recognise and understand so they can be learned from with future platforms, which is why I mentioned I would not dismiss GPUs from your future environment, as you now have very valuable first hand experience of the issues that can be faced, and it’s well worth putting that experience to good use in the next platform design.

Let’s see if anything gets released to complete the Ampere range, hopefully they’ll have something with more than 16GB that will make things easier for higher density designs.

Regards

MG