Hello! I am beginner user on Jetson NX (JetsonPack 4.5).
I have several (maybe simple) questions which I cannot find on Internet.
I have 2 classification CNN models (about 10 M parameters) are written on Python + TF 2.0.
I would like to use them on Jetson NX.
When I converted my models using TF converter these models work ok, but the loading process on Jetson NX is veru slow! It is about 400 sec! Is it ok for this device?? In this case, no any opportunity to fast reload model while in work process… It is very bad.
Maybe some one know and tell me what must I do else?
Another way that I found on Internet is to convert TF model to ONNX format (tf2onn packagex), then we can save engine and usi it by pycuda or onnx-tensorrt.
I spent a lot of time and installed tf2onnx. I understood how to convert TF model to onnx format and inference it using onnxruntime.
But the speed of loading model still slow. In other Internet place I found how to create and save rt engine. It loads very quick. But I do not know absolutely how to infer this engine.
I found that we can use pycuda. But it does not install on JetSon 4.5. A lot of erros. I used this command:
I found that we can use onnx-tensorrt. But it requires more new cmake utility (I updated it) and then the install process of onnx-tensorrt gves me a lot of erros, too.
Why is it so hard to use this device in production? Why is it absent to simple instruction how to cnvert the model in RTT format and inference it?
Please, help me and tell me what can I do for solving this problems?
The command sudo nvpmodel -m 0 turn on the mode with 15W2CORE. I use 15W6CORE. Wether I understand correct that in this mode we will get the faster estimations for TensorRT models?
This command (pip3 install numpy pycuda --user) gives me the next errors:
pip3 install pycuda --user
Collecting pycuda
Using cached pycuda-2021.1.tar.gz (1.7 MB)
Installing build dependencies … error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yivouqca/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i Simple index – setuptools wheel ‘numpy; python_version >= ‘"’“‘3.10’”’“‘’ 'numpy==1.19.5; python_version >= '”‘“‘3.8’”’“’ and python_version < '”‘“‘3.10’”’“‘’ 'numpy==1.15.4; python_version >= '”‘“‘3.7’”’“’ and python_version < '”‘“‘3.8’”’“‘’ 'numpy==1.12.1; python_version < '”‘“‘3.7’”’“‘’
cwd: None
Complete output (2488 lines):
Ignoring numpy: markers ‘python_version >= “3.10”’ don’t match your environment
Ignoring numpy: markers ‘python_version >= “3.8” and python_version < “3.10”’ don’t match your environment
Ignoring numpy: markers ‘python_version >= “3.7” and python_version < “3.8”’ don’t match your environment
Collecting setuptools
Using cached setuptools-57.2.0-py3-none-any.whl (818 kB)
Collecting wheel
Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting numpy==1.12.1
Using cached numpy-1.12.1.zip (4.8 MB)
Building wheels for collected packages: numpy
Building wheel for numpy (setup.py): started
Building wheel for numpy (setup.py): still running…
Building wheel for numpy (setup.py): still running…
Building wheel for numpy (setup.py): still running…
Building wheel for numpy (setup.py): finished with status ‘error’
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '”‘"’/tmp/pip-install-gt4iqm1r/numpy_4ffb26f2b4e3445eb0bec93cb49f69dc/setup.py’“'”‘; file=’“'”‘/tmp/pip-install-gt4iqm1r/numpy_4ffb26f2b4e3445eb0bec93cb49f69dc/setup.py’“'”‘;f = getattr(tokenize, ‘"’“‘open’”’“‘, open)(file) if os.path.exists(file) else io.StringIO(’”‘“‘from setuptools import setup; setup()’”’“‘);code = f.read().replace(’”‘"’\r\n’“'”‘, ‘"’"’\n’“'”‘);f.close();exec(compile(code, file, ‘"’“‘exec’”’"‘))’ bdist_wheel -d /tmp/pip-wheel-ig5n9soi
cwd: /tmp/pip-install-gt4iqm1r/numpy_4ffb26f2b4e3445eb0bec93cb49f69dc/
Complete output (2152 lines):
Running from numpy source directory.
blas_opt_info:
blas_mkl_info:
libraries mkl_rt not found in [‘/usr/local/lib’, ‘/usr/lib’, ‘/usr/lib/aarch64-linux-gnu’]
NOT AVAILABLE
Although mode 0 only enables two CPUs, the clocks rate is much higher.
Since TensorRT mainly uses GPU for inference, the boost in CPU will help to launch kernel faster.
2. We don’t meet a similar error when installing pyCUDA.
It looks like the error is mainly related to NumPy.
Is there anything weird when using the pyCUDA?
In fact, the engines work quick and results look like correct results. But the reload model is more slow process then inference.
For example, I take batch from 8 images and get the inference time about 0.10 sec. for my CNN (it is abput 10 M parameters, 300x300 in 16bit). But if I want to reload another model it can take abput 0.25-0.30 sec. Is it ok? Maybe is theere a way to improve the loading tiime?
May I know which frameworks do you use for inference currently? TensorFlow or TensorRT?
If TensorRT is used, have you tried to serialize the engine or convert it every time from an ONNX model?
Serialization can save you some conversion time which is triggered in the initial.