Please provide the following info (tick the boxes after creating this topic):
Submission Type
Bug or Error
Feature Request
Documentation Issue
Question
Other
Workbench Version
Desktop App v0.44.8
CLI v0.21.3
Other
Host Machine operating system and location
Local Windows 11
Local Windows 10
Local macOS
Local Ubuntu 22.04
Remote Ubuntu 22.04
Other
The AI Workbench example readme’s document the memory size which is awesome . It is less awesome that none of the examples run on a single consumer card.
Can someone point me at a project that fits in 8gb, 12gb, 16gb or 20gb?
Are there any examples that run across multiple consumer cards in a machine.
Here is a full list of the example projects we offer.
The finetuning projects generally require more GPU compute, so it may be difficult to run those on a small consumer card (especially those dealing with NeMo Framework). The mistral finetuning project you may be able to fit as long as you run at lower quantization, eg. 4-bit. Additionally, the sdxl customization project may be able to run on a smaller consumer card as well, eg. 16gb or lower.
The Hybrid RAG project does not require a GPU if running with cloud endpoints; if you would like to inference with a model locally, you can quantize a smaller model down to 4 bits and fit it onto a 16gb or lower card.
The data science projects should be able to use smaller GPUs to accelerate libraries like pandas and sklearn.
You can assign multiple GPUs to a project under Environment > Hardware > GPUs. For many examples the default is 1, but this can be adjusted to assign more GPUs to a project at runtime.
Some README definitions may need updating but in general those are geared more towards recommended rather than minimum specs.
If working on a multi-GPU system, you can attach more GPUs to a project under Environment > Hardware. They should show up in the project container under nvidia-smi and you can use it to run models like any other multi GPU system.