I’ve been trying to work within the framework of AI Workbench, but it’s getting a bit frustrating. The latest annoyance is that there is something about how AI Workbench manages the containers on the server which leads to them being shut down if the AI Workbench client gets disconnected. This most often happens if I have connected to the Spark over a VPN or something, and the VPN disconnects.
This is very, very annoying, if the Spark was running a multi-day training or fine-tuning!
I now believe it’s time to start working directly within docker containers, ideally based on the latest ngc pytorch 2.9 container, and just build a persistent environment inside there that I launch from the command line inside a tmux or something.
Ideally, I would still have a simple way to run jupyter lab and VS Code into the container, is there any prior art for getting this running while still maybe using a Custom app in the nVidia Sync app or something like that, to port-forward to more persistent container-exposed ports on the spark?
Then maybe I could actually start recording epoch stats in Weights & Biases for example.
Any guidance appreciated!
hi simon - sorry for taking so long to respond.
we don’t deal well with long running jobs, and it’s in the backlog to address this issue.
this post will help me elevate the feature request.
Workbench is a bit like JupyterLab in that if the connection is broken, things will stop and you can’t just pick back up.
for example if the connection is broken or you close the location window, the location stops and stops the running container.
to some extent, the issue is having the environment instantiated in the workbench project container but only being able to work with that container through Workbench.
Not to throw another workbench solution at you, but you could try the following:
- Get things setup with a script in the project repo
- ssh into the remote
- Then run the script with a docker command.
I haven’t done this yet but I will give it a shot to see how it works.
Yeah I had to abandon AI Workbench months ago, for this, but more importantly, for lack of an AI Workbench Image with a recent pytorch.
Instead, I now have to manage my own docker image based on the NGC PyTorch container and manage all my own apt/pip dependencies myself in the Dockerfile.
understood.
we are working these issues out. when we’ve got them sorted, would be very appreciative if you could kick the tires again and let us know what you think.