Hello, I am using a jetson device for federated learning pytorch training, how can I call my GPU in a docker container to complete pytorch training?The docker image version I have installed is nvidia-jetson-l4t-ml-r35.1.0-py3 and my jetpack version is 5.1.0
The Pytorch package in the l4t-ml container has enabled CUDA support already.
So you can train it with GPU in the same way as other environments.
Thanks, I just found out that it works, and I think the problem may be with my federated learning framework
The federated learning framework I use is fedml, and I can successfully check the CUDA in Doker, but why is it really running IGPU during training.The instructions to start the container are:docker run -t -i --runtime nvidia fedml/fedml:latest-nvida-jetson-l4t-ml-r35.1.0-py3 /bin/bash.
Is the GPU usage measured when the training time?
If yes, it does use GPU but just not occupy all the resources.
A common cause is the task is memory-bound so GPU needs to wait for the data to compute.
Yes, I started training when I looked at JTOP and the usage of the right GPU was always a flat line during the whole process, but the IGPU occupation always kept a high number, and the training of Cifar10 image classification took two hours, and my RTX3060 graphics card only took thirty minutes, all I thought was just calling the core graphics card
Could you explain more about the IGPU you mean?
Nano is iGPU so the usage is high?
Moreover, we don’t expect Nano can achieve the same performance as a desktop GPU.
4 times slower sounds okay for Nano.
I’m very sorry, I chose the wrong post tag before, my device is Jetson AGX Xavier, according to the information I checked, the IGPU is a graphics integrated graphics card in the CPU, its performance is not as good as the discrete graphics card, so I think I only called the integrated graphics card, and did not successfully call the discrete graphics card.
Do you mean the training works on iGPU (Jetson) but not work on dGPU (3060)?
If yes, we can move your topic to the desktop GPU board.