Expected performance of the Jetson Orin NX 16GB

Abo_Jaafar · July 22, 2023, 1:52pm

Hello,

I have a couple of questions concerning the expected performance of this device:
I am running a neural network on the original Jetson nano which takes roughly 800ms per image.
Now, I know that the Orin NX 16GB is advertised to have 100 TOPS of processing power, several orders of magnitude the processing power of the Nano, but I also know that performance cant be calculated linearly.
My question is, what performance should I expect from the Orin Nx 16gb on the same task? If I where to run the two instances of this task simultaneously , will I be able to achieve sub 250ms speeds per image on each instance? Can this also be achieved on the Orin Nx 8Gb ?

My other question concerns training a model on the Jetson Orin Nx.
I know that these systems are optimised for inference and perform poorly on training tasks.
However, if training needs to be done occasionally, what performance should I expect? I have a network being trained on 1050ti 4GB taking about 4-6h to complete, is there a way to know how would the Nx behave in such a scenario ? how long would it take?

I know that my questions are a little oblivious, but we’re working on a project and want to make sure that the chosen hardware is enough.

Anything helps, thanks in advance.

AastaLLL · July 24, 2023, 2:44am

Hi,

Performance usually depends on the type of task.
The behavior of IO-bound inference (ex. small kernel convolution) will be very different compared to computation-boud inference.

So it’s recommended to give it a try directly.
But we do have some benchmark results for Orin NX for your reference:

Thanks.

Abo_Jaafar · July 24, 2023, 6:33am

Thanks for the reply @AastaLLL and for the benchmarks.

Do you have any advice or comments on the training part of my question ?

Also I have another question:

Is there a recommended method or approach for parallel inference with python on the Orin NX ? Any best practices?

AastaLLL · July 26, 2023, 5:54am

Hi,

Here are some data about the training time of Jetson for your reference.
Usually, transfer learning a classification takes several hours to finish.

github.com

dusty-nv/jetson-inference/blob/master/docs/pytorch-transfer-learning.md#training-datasets

<img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/deep-vision-header.jpg" width="100%">
<p align="right"><sup><a href="depthnet.md">Back</a> | <a href="pytorch-cat-dog.md">Next</a> | </sup><a href="../README.md#hello-ai-world"><sup>Contents</sup></a>
<br/>
<sup>Transfer Learning</sup></s></p>

# Transfer Learning with PyTorch

Transfer learning is a technique for re-training a DNN model on a new dataset, which takes less time than training a network from scratch.  With transfer learning, the weights of a pre-trained model are fine-tuned to classify a customized dataset.  In these examples, we'll be using the <a href="https://arxiv.org/abs/1512.03385">ResNet-18</a> and [SSD-Mobilenet](pytorch-ssd.md) networks, although you can experiment with other networks too.

<p align="center"><a href="https://arxiv.org/abs/1512.03385"><img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/pytorch-resnet-18.png" width="600"></a></p>

Although training is typically performed on a PC, server, or cloud instance with discrete GPU(s) due to the often large datasets used and the associated computational demands, by using transfer learning we're able to re-train various networks onboard Jetson to get started with training and deploying our own DNN models.  

<a href=https://pytorch.org/>PyTorch</a> is the machine learning framework that we'll be using, and example datasets along with training scripts are provided to use below, in addition to a camera-based tool for collecting and labeling your own training datasets.  

## Installing PyTorch

If you are [Running the Docker Container](aux-docker.md) or optionally chose to install PyTorch back when you [Built the Project](building-repo-2.md#installing-pytorch), it should already be installed on your Jetson to use.  Otherwise, if you aren't using the container and want to proceed with transfer learning, you can install it now:

``` bash

This file has been truncated. show original

Thanks.

system · August 16, 2023, 6:27am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.