What is the cheapest and easiest way to run AODT with a VM online, and is there any good documentation on it.
I have used vast.ai and tensordock but ran into problems with both.
Let me know what you guys are using thats cheap and works, and easy to setup.
Also, the documentations are not that helpful. How does one go about connecting the VMs to run the backend and the frontend if I am following directions for this and not azure:
The azure docs are also very confusing as I don’t know where to run what (local, remote (if so, on what os? what machine (frontend or backend)?), etc. It does not specifically say where I have to run what, which makes it hard to follow the instructions.
Thanks.
@asinha6
"What is the cheapest and easiest way to run AODT "
We are unable to provide guidance on this. You can run AODT on any server (on premise or cloud) that meets the requirement specified in documentation.
“vast.ai and tensordock but ran into problems with both.”
Both should support AODT. Can you please elaborate on:
- What was the problem?
- What GPU did you use?
- What was the OS for front and backend
- Did the VM come preinstalled with drivers?
The expectation is that you start with a clean install on a VM that has only the OS installed.
“Let me know what you guys are using thats cheap and works, and easy to setup.”
We are asking our customers to use Azure and all of them have been successful. Azure is a single script installation-our script will provision the VMs, and set up both front and back ends
“How does one go about connecting the VMs to run the backend and the frontend if I am following directions for this and not azure”
We would need to create a custom installation specifically for your case. If you can elaborate on the above questions, I can provide some scripts that you can try
Thanks for your response,
I used 2 RTX 6000 Ada GPUs for a single machine (with Ubuntu 22.04) for both vast and tensordock. Both came with NVIDIA Drivers. Vast AI did not work because I was using a docker template already and so when I did:
./aodt_bundle/install.sh localhost $NGC_CLI_API_KEY
the error was something like "couldn’t find /dev/mem
Tensordock was being unreliable so that was the main issue but I brought it up because I was wondering if it’s possible to run AODT via these steps:
Get NoMachine to connect to the 2x RTX 6000 Ada VM and run the same installation steps as the ones under " [Dell R750]" on the AODT installation docs.
Btw, the only VM that had a GUI by default was the one from vast.ai, but I always got the “missing /dev/mem” error. Tensordock is CLI and NoMachine can work, it’s just that sometimes it does not let me connect to the VM for several days at a time. Tensordock will probably eventually work.
Thank you so much for responding once again.
@asinha6 Installation for R750 is the correct template. Our installation installs a VNC server so a GUI is not required. I assume you wish to enable the vast.ai deployment. Can you please share the linux console commands and response? I would like to see where in the installation you are seeing the “couldn’t find /dev/mem” error.
@kpasad1
I am in and (almost) everything works. It’s not letting me launch from the launcher.
When I click Launch it just loads for a little bit and just doesn’t launch anything. It says “Launching” for a couple of seconds and then goes back to saying “Launch”
I should mention it says this before the vnc server starts:
Here is what comes up when I do nvidia-smi:
And here are my drivers:
@kpasad1
1.
nvidia-logs.tar.gz (545.7 KB)
nvidia-bug-report.log.gz (548.3 KB)
- No, this is tensordock
- Using TigerVNC on my local windows box, doing myip:22030. 22030 is allocated to internal port 5901 on the VM. I just do localhost when running the ./install.sh command or just myip:22030 and both let me connect but never let me launch. When I do localhost specifically (./install.sh localhost $API_KEY) this always happens:
Also I should mention that I get this error:
“Error occurred during installation of : asim@1.0.0 is already installed.”
After this “preparing” completes:
@kpasad1 I am using the cli command to install so maybe that’s why its failing?
I’m using this to get the aodt_bundle.zip:
ngc registry resource download-version “esee5uzbruax/aodt-installer:1.0.0”
The curl command with the link in the docs always fails for me and it doesn’t download it correctly since it downloads instantly and never lets me unzip (even with the correct link, with seperating Bearer and $TOKEN).
Also, what is the “NGC_CLI_API_deploymentKEY”. That also may be why its not working? I am using a personal API key by the way.
Here are more logs in my /asim-1.0.0//data/Kit/aerial_sim/1.0/ folder:
1.0-20240802T155819Z-001.zip (12.9 KB)
Perhaps there’s a specific script or different commands for my case that sorts all of this out?
Thanks for all your help.
@asinha6 You are correct, the NGC_CLI_API_deploymentKEY should be replaced by NGC_CLI_API_KEY. Please also note the missing space between in the curl command between “Bearer” and “$Token”. The correct commands are:
export NGC_CLI_API_KEY=<NGC_CLI_API_KEY>
AUTH_URL=“https://authn.nvidia.com/token?service=ngc&scope=group/ngc:esee5uzbruax&group/ngc:esee5uzbruax/”
TOKEN=$(curl -s -u “$oauthtoken”:“$NGC_CLI_API” -H “Accept:application/json” “$AUTH_URL” | jq -r ‘.token’)
versionTag=“1.0.0”
downloadedZip=“$HOME/aodt_bundle.zip”
curl -L “https://api.ngc.nvidia.com/v2/org/esee5uzbruax/resources/aodt-installer/versions/$versionTag/files/aodt_bundle.zip” -H “Authorization: Bearer $TOKEN” -H “Content-Type: application/json” -o $downloadedZip
unzip -o $downloadedZip
Hi,
The AUTH_URL you gave above is wrong:
Same with TOKEN command:
And thus, the curl command does not work:
Note: I already tried sepearting the bearer and token before and it still download a “aodt_bundle.zip” that is not a zip file since it downloaded instantly and didn’t let me unzip.
@asinha6 Sorry, looks like inverted comma got switch to windows format (“) instead of ("). I am rewriting the instructions. Additionally:
- Check if a token is generated by doing:
echo $TOKEN.
TOKEN is an alphanumerical string. If it is empty, make sure the API Key is entered correctly. It is not uncommon to missing out the last charecter.
- Run the curl commands with -v option. The verbose will generate addtional data to assist in debug
export NGC_CLI_API_KEY=<NGC_CLI_API_KEY>
AUTH_URL=“https://authn.nvidia.com/token?service=ngc&scope=group/ngc:esee5uzbruax&group/ngc:esee5uzbruax/”
TOKEN=$(curl -s -u "\$oauthtoken":"$NGC_CLI_API_KEY" -H "Accept:application/json" "$AUTH_URL" | jq -r '.token')
versionTag="1.0.0"
downloadedZip="$HOME/aodt_bundle.zip"
curl -L "https://api.ngc.nvidia.com/v2/org/esee5uzbruax/resources/aodt-installer/versions/$versionTag/files/aodt_bundle.zip" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -o $downloadedZip
Hi,
when I do echo $TOKEN it is always empty. I am entering my API key as what you told me and in the docs:
export NGC_CLI_API_KEY=nvapi-**********5n0
Is this the correct way to enter it? Perhaps this is where I am doing something wrong? This is key is under “Personal Keys”
the “nvapi-” prefix should not be there. e.g.
export NGC_CLI_API_KEY=ZTU3***Zi
Generate the API key instead of personal key:
It worked!
Will this installation be different from the NGC CLI command I copied?
No, it is the exactly the same installation.
Alright.
I did ./install.sh localhost $NGC_CLI_API_KEY but it’s still not launching and I got the same error.
It’s a bit different this time because Im getting more information after the “The vncserver is now running. You can now connect using the vnc client”
It tells me my API_Key and the aerial_pass
Should I uninstall aerial from omniverse, unzip aodt_bundle.zip and then run the installation again?
Hi @kpasad1
I am going to uninstall everything and re-install it to see if it works. I have been trying to get Aerial work for several weeks now so if re-instaling does not work, can you provide the scripts you were talking about earlier to make Aerial run for my case? My lab would like to see progress as soon as possible and we really need Aerial to work in order to do so.
Thanks.
@kpasad1
Big update: it has launched but I only see a gray screen.
