Hello! I am beginner user on Jetson NX (JetsonPack 4.5).
I have several (maybe simple) questions which I cannot find on Internet.
I have 2 classification CNN models (about 10 M parameters) are written on Python + TF 2.0.
I would like to use them on Jetson NX.
- When I converted my models using TF converter these models work ok, but the loading process on Jetson NX is veru slow! It is about 400 sec! Is it ok for this device?? In this case, no any opportunity to fast reload model while in work process… It is very bad.
Maybe some one know and tell me what must I do else?
- Another way that I found on Internet is to convert TF model to ONNX format (tf2onn packagex), then we can save engine and usi it by pycuda or onnx-tensorrt.
I spent a lot of time and installed tf2onnx. I understood how to convert TF model to onnx format and inference it using onnxruntime.
But the speed of loading model still slow. In other Internet place I found how to create and save rt engine. It loads very quick. But I do not know absolutely how to infer this engine.
I found that we can use pycuda. But it does not install on JetSon 4.5. A lot of erros. I used this command:
sudo pip3 install --global-option=build_ext --global-option="-I/usr/local/cuda/include" --global-option="-L/usr/local/cuda/lib64" pycuda
I found that we can use onnx-tensorrt. But it requires more new cmake utility (I updated it) and then the install process of onnx-tensorrt gves me a lot of erros, too.
Why is it so hard to use this device in production? Why is it absent to simple instruction how to cnvert the model in RTT format and inference it?
Please, help me and tell me what can I do for solving this problems?