YOLO v3 crashed on Nvidia Jetson Xavier NX

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
#1

Hi:

I have Nvidia Jetpack 4.5.1 installed on my Jetson Xavier NX dev kit. It has CUDA 10.2, CUDNN 8.0. I am trying to run YOLO v3 (GitHub - pjreddie/darknet: Convolutional Neural Networks) If I build the CPU version, I can run the demo. However, if I build the GPU version (GPU=1, CUDNN=0), it crashed when loading the weights as shown below. How can I fix it?

nvidia@XavierNx:~/apps/darknet$ ./darknet detect cfg/yolov3.cfg yolov3.weights data/dog.jpg
layer filters size input output
0 1 2 3 4 res 1 304 x 304 x 64 → 304 x 304 x 64
5 6 7 8 res 5 152 x 152 x 128 → 152 x 152 x 128
9 10 11 res 8 152 x 152 x 128 → 152 x 152 x 128
12 13 14 15 res 12 76 x 76 x 256 → 76 x 76 x 256
16 17 18 res 15 76 x 76 x 256 → 76 x 76 x 256
19 20 21 res 18 76 x 76 x 256 → 76 x 76 x 256
22 23 24 res 21 76 x 76 x 256 → 76 x 76 x 256
25 26 27 res 24 76 x 76 x 256 → 76 x 76 x 256
28 29 30 res 27 76 x 76 x 256 → 76 x 76 x 256
31 32 33 res 30 76 x 76 x 256 → 76 x 76 x 256
34 35 36 res 33 76 x 76 x 256 → 76 x 76 x 256
37 38 39 40 res 37 38 x 38 x 512 → 38 x 38 x 512
41 42 43 res 40 38 x 38 x 512 → 38 x 38 x 512
44 45 46 res 43 38 x 38 x 512 → 38 x 38 x 512
47 48 49 res 46 38 x 38 x 512 → 38 x 38 x 512
50 51 52 res 49 38 x 38 x 512 → 38 x 38 x 512
53 54 55 res 52 38 x 38 x 512 → 38 x 38 x 512
56 57 58 res 55 38 x 38 x 512 → 38 x 38 x 512
59 60 61 res 58 38 x 38 x 512 → 38 x 38 x 512
62 63 64 65 res 62 19 x 19 x1024 → 19 x 19 x1024
66 67 68 res 65 19 x 19 x1024 → 19 x 19 x1024
69 70 71 res 68 19 x 19 x1024 → 19 x 19 x1024
72 73 74 res 71 19 x 19 x1024 → 19 x 19 x1024
75 76 77 78 79 80 81 82 yolo
83 route 79
84 85 upsample 2x 19 x 19 x 256 → 38 x 38 x 256
86 route 85 61
87 88 Killed

#2

Try rebuilding with CUDNN=1, I believe when enabled with GPU they use Cudnn as well.

Thank you

#3

I also crashed when I compiled with CUDNN=1

#4

It would be great if you can post a screenshot of your terminal. Thank you

#5

Hello,
Did you modify this line:
ARCH= -gencode arch=compute_30,code=sm_30
-gencode arch=compute_35,code=sm_35
-gencode arch=compute_50,code=[sm_50,compute_50]
-gencode arch=compute_52,code=[sm_52,compute_52]

if not replace that with

ARCH=-gencode arch=compute_72,code=sm_72

This is just adding the proper CUDA arch number for Jetson Xavier NX. Try this if not please share the screenshot. Thanks

#7

After change the line, still the same

#8

pls share your Makefile

#9

Makefile (3.0 KB)

#10

Hi,

Usually ‘killed’ is caused by out-of-memory.
Could you monitor your device with tegrastats and share the output log with us?

Thanks.

#11

I agree with AastaLLL,

  1. Makefile

GPU=1
CUDNN=1

  1. modify cfg/yolov3.cfg

batch=1
subvisions=1
…

  1. tiny

nvidia@XavierNx:~/apps/darknet$ ./darknet detect cfg/yolov3-tiny.cfg yolov3-tiny.weights data/dog.jpg

#12

You are right. The cfg file was set to tranning mode. After I change it to

batch=1
subvisions=1

It worked fine.

Thank you very much.

#13

However, if I build with

CUDNN=1

It crashed when loading the weight.

It worked if I use YoLo V3 tiny, even for CUDNN. But I really need YOLO, not the tiny.

#14

^.^

G’luck

#15

Has anyone run Yolo V3 inference with GPU=1 CUDNN=1 on Xavier NX successfully?

#17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.