I’m trying to train a network on my Nano. I know it’s pushing it’s capabilities, but I’m curious to try.
I’ve got an SSD drive and have set it so that I have a large swap, but when I try to train a network I get OOM errors. It looks to me that the Nano’s Tegra chip doesn’t use the swap, so the chip runs out of memory.
So, what I think is happening is that the regular OS uses the swap for regular non-gpu memory, but the Tegra doesn’t. Is that correct? Is there any workaround?
The OOM is from the GPU, not ram. Tegra is an edge device basically and has only 4GB of on-card memory, which is very small amount for training.
If you really want to try doing that, reduce the batch size in your training - it is probably set for cards with 12GB so try halving and if still oom, halve again.
However, training with very small batch size is a problem by itself…
Some devices/software can use virtual memory (which can use swap), but other devices need physical memory. You would be correct to say the GPU cannot use swap.
Swap still has some use since it means other programs can be swapped out and leave more physical memory for the GPU, but there is a limit to how much that will save. Looks like you’ve gone past that limit.
You can indeed use Swap on the Nano - it was one of the first few things I did to the Nano after setting it up.
After adding an SSD to the Nano, then getting the filesystem off my mSD card, I then went for the Swap. There are two key videos on YouTube that explain the process pretty well.
The Channel is JetsonHacks[dot]com. There is a video on ‘Pivoting the Root,’ then a second one where swap is added. It worked pretty well for me; & the dude runs memtester to show the swap is working as it should.
@jj216216: Could you extend how do you connect SSD to the Nano, please?
@raspi.three You got it! That is the site & the YouTube video I used to set it up - that guy is a great help!