Segmentation Fault (core dumped) occurs when YOLOX is converted to TensorRT

[vinnrotich]

I tried to convert YOLOX-nano.pth model to TensorRT with Jetson nano developer kit but got a segmentation fault.
I tried the same on desktop successfully.

python3 trt.py -f /home/jetson/yolox_nano/YOLOX/exps/example/custom/yolox_nano.py -c /home/jetson/yolox_nano/YOLOX/YOLOX_outputs/yolox_nano/best_ckpt.pth 

Jetson nano 4GB
JetPack4.6, Cuda10.2
CuDNN 8.2
TensorRT 8.0.1.6
Python3.6.9
OS: Ubuntu18.04LTS
Torch1.9.0

I have checked all the previous raised questions, but all the solutions dont work.
By the way the issue is also raised here segmentation fault

[AastaLLL]

Hi,

Would you mind monitoring your device status with tegrastats first.
Especially to check if an out-of-memory error occurs.

$ sudo tegrastats

Thanks.

[vinnrotich]

I did

RAM 2334/3956MB (lfb 3x4MB) SWAP 774/1978MB (cached 49MB) IRAM 0/252kB(lfb 252kB) CPU [21%@307,14%@307,23%@307,10%@307] EMC_FREQ 4%@1600 GR3D_FREQ 6%@537 APE 25 PLL@22C CPU@25C PMIC@50C GPU@22C AO@29.5C thermal@23.5C POM_5V_IN 2438/2911 POM_5V_GPU 121/157 POM_5V_CPU 365/704

[AastaLLL]

Hi,

Sorry for the late update.
It seems that the memory is enough when running the script.

Is there any concurrent memory access on your code?
Please noted that concurrent access of a GPU buffer is only supported on the dGPU system.

Thanks.

[naisy]

I tried this with Nano 2GB/4GB because it is similar to the problem I have been trying with JetBot JetPack 4.6/4.6.1 for the past month.
The result was that both Nano 2GB/4GB worked.

JetBot image: EDIT: Use 64GB SDCARD
2GB Nano: jetbot-nano-2gb-jp461-2022-04-04.img.zip - Google Drive
4GB Nano: jetbot-nano-4gb-jp461-2022-04-04.img.zip - Google Drive

Open the JupyterLab Terminal in your browser, just like JetBot.
Execute the following commands.

# Install YOLOX
cd ~/github
git clone https://github.com/Megvii-BaseDetection/YOLOX
cd YOLOX
mv requirements.txt R.txt
touch requirements.txt

pip install -e .
pip install loguru
pip install thop
pip install tabulate
pip install pycocotools
pip install tqdm

# download pre-trained yolox_nano.pth model
wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_nano.pth

# build engine
time python3 ./tools/trt.py -f ./exps/default/yolox_nano.py -c yolox_nano.pth 
ls YOLOX_outputs/yolox_nano/

# infer
python3 tools/demo.py image -f ./exps/default/yolox_nano.py --trt --save_result

# copy to the ~/Notebooks
cp -r ./YOLOX_outputs ~/Notebooks

This is the screen shot when it is executed.

Memory usage:

Output image:

I am now running Object Following on a JetBot 2GB/4GB and am really surprised at how well it works.

One special setting for this image is the ZRAM setting.
reference: Linux Performance: Almost Always Add Swap. Part 2: ZRAM

sudo bash -c 'echo "vm.vfs_cache_pressure=500" >> /etc/sysctl.conf'
sudo bash -c 'echo "vm.swappiness=100" >> /etc/sysctl.conf'
sudo bash -c 'echo "vm.dirty_background_ratio=1" >> /etc/sysctl.conf'
sudo bash -c 'echo "vm.dirty_ratio=50" >> /etc/sysctl.conf'
sudo reboot

And one more thing, I have built TensorRT 8.2.3 to install onnx_graphsurgeon-0.3.15.

[danielandrebach]

@naisy

One special setting for this image is the ZRAM setting.
reference: https://haydenjames.io/linux-performance-almost-always-add-swap-part2-zram/

sudo bash -c 'echo "vm.vfs_cache_pressure=500" >> /etc/sysctl.conf'
sudo bash -c 'echo "vm.swappiness=100" >> /etc/sysctl.conf'
sudo bash -c 'echo "vm.dirty_background_ratio=1" >> /etc/sysctl.conf'
sudo bash -c 'echo "vm.dirty_ratio=50" >> /etc/sysctl.conf'
sudo reboot

And one more thing, I have built TensorRT 8.2.3 to install onnx_graphsurgeon-0.3.15.

Thanks for your help.