Vi-output, gala: syncpoint id <any> stuck waiting

richsheep · June 1, 2021, 6:12am

Using jetson nano with custom carrier board,
sometimes, when using vi driver for image capture, no data received from v4l2 driver,

at the same time, dmesg shows:

[ 7294.450850] host1x 50000000.host1x: vi-output, gala: syncpoint id 17 (vi) stuck waiting 13, timeout=-1502
[ 7294.461400] ---- syncpts ----
[ 7294.461439] id 8 (gm20b_507) min 7792 max 7792 refs 1 (previous client : )
[ 7294.461457] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[ 7294.461500] id 11 (gm20b_505) min 1376 max 1376 refs 1 (previous client : gm20b_505)
[ 7294.461519] id 12 (gm20b_500) min 41024 max 41023 refs 1 (previous client : vi)
[ 7294.461538] id 13 (gm20b_501) min 40442 max 40441 refs 1 (previous client : vi)
[ 7294.461556] id 14 (gm20b_502) min 1760 max 1760 refs 1 (previous client : gm20b_504)
[ 7294.461600] id 15 (gm20b_503) min 14 max 14 refs 1 (previous client : gm20b_503)
[ 7294.461619] id 16 (gm20b_504) min 18 max 18 refs 1 (previous client : gm20b_502)
[ 7294.461637] id 17 (vi) min 12 max 13 refs 1 (previous client : gm20b_501)
[ 7294.461657] id 18 (vi) min 24 max 25 refs 1 (previous client : gm20b_500)


[ 7300.594060] host1x 50000000.host1x: vi-output, gala: syncpoint id 17 (vi) stuck waiting 13, timeout=-3002
[ 7300.603640] ---- syncpts ----
[ 7300.603651] id 8 (gm20b_507) min 7792 max 7792 refs 1 (previous client : )
[ 7300.603655] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[ 7300.603660] id 11 (gm20b_505) min 1376 max 1376 refs 1 (previous client : gm20b_505)
[ 7300.603666] id 17 (vi) min 12 max 13 refs 1 (previous client : gm20b_501)
[ 7306.737888] host1x 50000000.host1x: vi-output, gala: syncpoint id 17 (vi) stuck waiting 13, timeout=-4502
[ 7306.747465] ---- syncpts ----
[ 7306.747476] id 8 (gm20b_507) min 7792 max 7792 refs 1 (previous client : )
[ 7306.747480] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[ 7306.747485] id 11 (gm20b_505) min 1376 max 1376 refs 1 (previous client : gm20b_505)
[ 7306.747492] id 17 (vi) min 12 max 13 refs 1 (previous client : gm20b_501)
[ 7306.747496] id 18 (vi) min 24 max 25 refs

I wonder why vi-output stuck waiting timeout,

this only happened when using a python/pyQt5 application, C++ application cannot produce this error.

any solutions or suggestions?

much appreciated

ShaneCCC · June 1, 2021, 7:15am

Did you show the captured image by display or any others image process?

richsheep · June 2, 2021, 3:40am

nope, i using tensorrt before acquisition

ShaneCCC · June 2, 2021, 3:47am

Could you remove tensorrt to narrow down the problem.

richsheep · June 2, 2021, 3:52am

i removed tensorrt and then:

[ 4952.696826] host1x 50000000.host1x: vi-output, gala: syncpoint id 12 (vi) stuck waiting 1160, timeout=-1502
[ 4952.706741] ---- syncpts ----
[ 4952.706755] id 8 (gm20b_507) min 27236 max 27236 refs 1 (previous client : )
[ 4952.706760] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[ 4952.706766] id 11 (gm20b_505) min 35786 max 35786 refs 1 (previous client : gm20b_505)
[ 4952.706772] id 12 (vi) min 1159 max 1160 refs 1 (previous client : vi)
[ 4952.706777] id 13 (vi) min 1159 max 1160 refs 1 (previous client : vi)
[ 4952.706782] id 14 (gm20b_504) min 155834 max 155836 refs 1 (previous client : gm20b_504)
[ 4952.706787] id 15 (gm20b_503) min 96454 max 96454 refs 1 (previous client : gm20b_503)
[ 4952.706793] id 16 (gm20b_502) min 32998 max 33000 refs 1 (previous client : gm20b_502)
[ 4952.706798] id 17 (gm20b_501) min 6346 max 6346 refs 1 (previous client : gm20b_501)
[ 4952.706803] id 18 (gm20b_500) min 1580 max 1580 refs 1 (previous client : gm20b_500)

when waiting syncpoint 12, my application can acquire image from vi

seems i should not build tensorrt engine before acquisition?

ShaneCCC · June 2, 2021, 4:48am

Sorry too less information to me to comment this problem.

richsheep · June 2, 2021, 5:05am

Sorry about my unclear description

env:
Jetson Nano P3448-0002
Jetpack ver: 4.4
Kernel ver: 4.9

My conclusion now:

If run tensorrt engine at first, then start vi acquisition
syncpoint id of vi: 17 & 18
result: vi always stuck timeout, cannot getting any image data
If start vi acquisition at first, then run tensorrt engine
syncpoint id of vi: 12 & 13
result: OK with getting image data.

In my own application, I cannot guarantee tensorrt run after acquisition

My problem:

Why cannot getting image from vi, when vi syncpoint at 17 & 18?

How should I using tensorrt and vi in correct way?

ShaneCCC · June 2, 2021, 5:37am

Does any gm20b message when start the tensorrt engine before VI acpuisition?

richsheep · June 2, 2021, 7:14am

No message or dmesg prints occurred when start tensorrt engine before VI acquisition.

TensorRT running well.

richsheep · June 6, 2021, 11:24am

New test results:

This problem appear in following steps:

Run a GPU task

# Run a long time GPU task to occupy syncpt, 
# in my case: GPU will using 12, 13, 14, 15, 16
$ ./trtexec --onnx=model.onnx --warmUp=200000

Run acquisition

$ ./my-app
get image timeout!
get image timeout!

In dmesg:

[ 2014.104042] host1x 50000000.host1x: vi-output, gala: syncpoint id 17 (vi) stuck waiting 6, timeout=-1502
[ 2014.113987] ---- syncpts ----
[ 2014.114083] id 8 (gm20b_507) min 9248 max 9248 refs 1 (previous client : )
[ 2014.114133] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[ 2014.114185] id 11 (gm20b_505) min 624 max 624 refs 1 (previous client : gm20b_505)
[ 2014.114233] id 12 (gm20b_504) min 1075 max 1072 refs 1 (previous client : vi)
[ 2014.114278] id 13 (gm20b_503) min 401 max 398 refs 1 (previous client : vi)
[ 2014.114322] id 14 (gm20b_502) min 2 max 2 refs 1 (previous client : )
[ 2014.114367] id 15 (gm20b_501) min 2 max 2 refs 1 (previous client : )
[ 2014.114400] id 16 (gm20b_500) min 8 max 8 refs 1 (previous client : )
[ 2014.114420] id 17 (vi) min 5 max 6 refs 1 (previous client : vi)
[ 2014.114438] id 8 (vi) min 5 max 6 refs 1 (previous client : vi)

My current solution 1:

Stop GPU task process（stop using syncpoint 12, 13, 14, 15, 16）
Run acquisition again

$ ./my-app
get image success!
get image success!

My current solution 2:

Run acquisition again and again
After several times restart acquisition, get image will not timeout.
But, if stop GPU task after it fixed,
then restart acquisition, timeout will ouccr on syncpoint 12

[ 2065.556794] host1x 50000000.host1x: vi-output, gala: syncpoint id 12 (vi) stuck waiting 1172, timeout=-1502
[ 2065.566992] ---- syncpts ----
[ 2065.567085] id 8 (gm20b_507) min 9252 max 9252 refs 1 (previous client : )
[ 2065.567133] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[ 2065.567182] id 11 (gm20b_505) min 632 max 632 refs 1 (previous client : gm20b_505)
[ 2065.567228] id 12 (vi) min 1171 max 1172 refs 1 (previous client : vi)
[ 2065.567273] id 13 (vi) min 407 max 408 refs 1 (previous client : vi)
[ 2065.567331] id 17 () min 12 max 10 refs 0 (previous client : vi)
[ 2065.567366] id 18 () min 12 max 10 refs 0 (previous client : vi)

current solution cannot be implement in my use case.

Any other solutions? or have any suggestion ?

Many thanks.

ShaneCCC · June 7, 2021, 2:26am

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Could you attached the binary here to try.

system · June 25, 2021, 1:00am

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.