Hello,
We are testing an IMX264 sensor with Jetson TX1.
We are facing the following issue, any help is welcome.
root@tegra-ubuntu:/home/nvidia# export enableCamPclLogs=5
root@tegra-ubuntu:/home/nvidia# export enableCamScfLogs=5
root@tegra-ubuntu:/home/nvidia# /usr/sbin/nvcamera-daemon&
[1] 1204
root@tegra-ubuntu:/home/nvidia# gst-launch-1.0 nvcamerasrc fpsRange="20 20" ! fakesink
Setting pipeline to PAUSED ...
Thread 1 getting next capture
Thread 1 is waiting
Thread 2 getting next capture
Thread 2 is waiting
Thread 3 getting next capture
Thread 3 is waiting
Thread 5 getting next capture
Thread 5 is waiting
Thread 4 getting next capture
Thread 4 is waiting
Thread 6 getting next capture
Thread 6 is waiting
Thread 7 getting next capture
Thread 7 is waiting
Thread 8 getting next capture
Thread 8 is waiting
Thread 9 getting next capture
Thread 9 is waiting
Thread 10 getting next capture
Thread 10 is waiting
Thread 11 getting next capture
Thread 11 is waiting
Thread 12 getting next capture
Thread 12 is waiting
Starting services...
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread CaptureScheduler checkFramePending start
Worker thread CaptureScheduler frameStart start
Worker thread V4L2CaptureScheduler checkCaptureComplete start
Worker thread V4L2CaptureScheduler issueCaptures start
SCF: Error NotSupported: No EGL device available (in src/services/gl/GLService.cpp, function initialize(), line 139)
SCF: Error NotSupported: (propagating from src/services/gl/GLService.cpp, function startService(), line 49)
SCF: Error NotSupported: (propagating from src/components/ServiceHost.cpp, function startServices(), line 130)
SCF: Error NotSupported: (propagating from src/api/CameraDriver.cpp, function initialize(), line 153)
SCF: Error NotSupported: (propagating from src/api/CameraDriver.cpp, function getCameraDriver(), line 100)
Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
ERROR: Pipeline doesn't want to pause.
Setting pipeline to NULL ...
Freeing pipeline ...
[1]+ Segmentation fault /usr/sbin/nvcamera-daemon
Does the v4l2-ctl worked? You need to make sure the v4l2-ctl can capture the raw data first. And below is some debug information.
https://elinux.org/Jetson_TX2/28.1_Camera_BringUp
Yes the following code works and produces a mydata.raw file.
v4l2-ctl -d /dev/video0 -w --verbose --set-fmt-video=width=2448,height=2048,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=50 --stream-to=mydata.raw
But all images are black.
To enable trace for more information
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace
The following files aren’t found, (we’re working with 28.1 on TX1):
/sys/kernel/debug/tracing/events/tegra_rtcpu/enable
/sys/kernel/debug/tracing/events/freertos/enable
/sys/kernel/debug/camrtc/log-level
but there is:
/sys/kernel/debug/tracing/events/tegra_rtc/enable
Finally, we found out that from mydata.raw file, we could extract bayer frames but only 1 each 4 frames, all other in between being black. Any idea why is that?
And why nvcamerasrc fails if v4l2-ctl works?
Thanks
You need sudo su before those command. If only 1 each 4 frames that doesn’t mean works for v4l2-ctl.
olhx
July 20, 2018, 9:28am
7
Hello ShaneCCC.
I am a colleague of Ketchup, we work on the same platform. I have made the same test, using “su”,
with the same observations. Even with the commands from comment #4 , dated 2018-07-13 05:47, we can
not observe more than what Ketchup has shown in comment #1 , dated 2018-07-10 09:33.
Do we need to enable something to better observe the kernel? We lack observability to understand
why “nvcamera-daemon” crashes.
By the way: when placing the same IMX264 sensor on the Jetson TX1 evaluation kit, with the same kernel,
“nvcamera-daemon” does not crash.
[ 2625.473006] nvcamera-daemon[1300]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x92000006
[ 2625.483729] pgd = ffffffc0f885c000
[ 2625.487232] [00000000] *pgd=0000000175768003, *pud=0000000175768003, *pmd=0000000000000000
[ 2625.495772]
[ 2625.497335] CPU: 0 PID: 1300 Comm: nvcamera-daemon Tainted: G W 4.4.38-qcam5-g3e05b14 #1
[ 2625.506595] Hardware name: jetson_tx1 (DT)
[ 2625.510736] task: ffffffc07afba580 ti: ffffffc07221c000 task.ti: ffffffc07221c000
[ 2625.518277] PC is at 0x402e0c
[ 2625.521277] LR is at 0x401ef8
[ 2625.524348] pc : [<0000000000402e0c>] lr : [<0000000000401ef8>] pstate: 80000000
[ 2625.532570] sp : 0000007facb492d0
[ 2625.535906] x29: 0000007facb4d9d0 x28: 0000007facb4e1e0
[ 2625.541305] x27: 0000000000000002 x26: 0000007facb4d340
[ 2625.546682] x25: 0000000000404000 x24: 0000000000000334
[ 2625.552057] x23: 0000007facb4b340 x22: 0000007facb4a340
[ 2625.557540] x21: 0000007facb4b264 x20: 0000007facb49350
[ 2625.563158] x19: 0000007facb4b950 x18: 0000000000000014
[ 2625.568568] x17: 0000007fae3bd760 x16: 0000000000415970
[ 2625.573946] x15: 0000007faed9ccc0 x14: 0000000000000000
[ 2625.579323] x13: 0000000000000008 x12: a3d70a3d70a3d70b
[ 2625.584725] x11: 0000007facb490b0 x10: 0101010101010101
[ 2625.590196] x9 : 0000000000000004 x8 : 00000000000000cf
[ 2625.595839] x7 : 0000000000000000 x6 : 0000007facb4929c
[ 2625.601233] x5 : 0000007facb4e2e8 x4 : 0000007facb4e1e0
[ 2625.606605] x3 : 0000000000401f30 x2 : 0000000000402e04
[ 2625.611978] x1 : 0000000000000001 x0 : 0000000000000000
[ 2625.617540]
[ 2625.619145] Library at 0x402e0c: 0x400000 /usr/sbin/nvcamera-daemon
[ 2625.626560] Library at 0x401ef8: 0x400000 /usr/sbin/nvcamera-daemon
[ 2625.634967] vdso base = 0x7faed9b000
...
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread CaptureScheduler checkFramePending start
Worker thread CaptureScheduler frameStart start
Worker thread V4L2CaptureScheduler checkCaptureComplete start
Worker thread V4L2CaptureScheduler issueCaptures start
SCF: Error NotSupported: No EGL device available (in src/services/gl/GLService.cpp, function initialize(), line 139)
SCF: Error NotSupported: (propagating from src/services/gl/GLService.cpp, function startService(), line 49)
SCF: Error NotSupported: (propagating from src/components/ServiceHost.cpp, function startServices(), line 130)
SCF: Error NotSupported: (propagating from src/api/CameraDriver.cpp, function initialize(), line 153)
SCF: Error NotSupported: (propagating from src/api/CameraDriver.cpp, function getCameraDriver(), line 100)
Follow below to get the trace log to check.
sudo su
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
v4l2-ctl xxxxxx
cat /sys/kernel/debug/tracing/trace
olhx
July 20, 2018, 10:03am
9
As explained by Ketchup, 3 sysfs control files are not present. Here it the output and kernel log:
$ sudo su
# systemctl stop nvcamera-daemon
# export enableCamPclLogs=5
# export enableCamScfLogs=5
# echo 1 > /sys/kernel/debug/tracing/tracing_on
# echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
# echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
bash: /sys/kernel/debug/tracing/events/tegra_rtcpu/enable: No such file or directory
# echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
bash: /sys/kernel/debug/tracing/events/freertos/enable: No such file or directory
# echo 2 > /sys/kernel/debug/camrtc/log-level
bash: /sys/kernel/debug/camrtc/log-level: No such file or directory
# echo > /sys/kernel/debug/tracing/trace
# v4l2-ctl -d /dev/video0 --set-fmt-video=width=2560,height=2048,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=20 --stream-to=test.raw
<<<<<< 4.58 fps
<<<<< 4.76 fps
<<<<< 4.68 fps
<<<< 4.69 fps
[Here "v4l2-ctl" hangs indefinitely]
[ 336.195844] video4linux video0: MW_ACK_DONE syncpoint time out!0
[ 336.202054] Unhandled fault: alignment fault (0x96000061) at 0xffffff8000000021
[ 336.209446] Internal error: : 96000061 [#1] PREEMPT SMP
[ 336.214702] Modules linked in: bcmdhd bluedroid_pm [last unloaded: fpga_mgr]
[ 336.221847] CPU: 2 PID: 1094 Comm: v4l2-ctl Tainted: G W 4.4.38-qcam5-g3e05b14 #1
[ 336.230473] Hardware name: jetson_tx1 (DT)
[ 336.234581] task: ffffffc0f62c0000 ti: ffffffc07d6d4000 task.ti: ffffffc07d6d4000
[ 336.242113] PC is at cil_write+0xc/0x14
[ 336.245970] LR is at tegra_csi_error_recover+0xf8/0x1e4
[ 336.251202] pc : [<ffffffc0007b6d14>] lr : [<ffffffc0007b77a0>] pstate: 60000145
[ 336.258609] sp : ffffffc07d6d7a10
[ 336.261930] x29: ffffffc07d6d7a10 x28: 0000000000000068
[ 336.267275] x27: 0000000000000014 x26: ffffffc07d755898
[ 336.272619] x25: 0000000000000214 x24: 0000000000000000
[ 336.277963] x23: 0000000000000020 x22: 0000000000000001
[ 336.283307] x21: ffffffc07d755900 x20: 0000000000000000
[ 336.288650] x19: ffffffc0f90cd018 x18: 0000000000000000
[ 336.293993] x17: 0000000000000000 x16: 0000000000000000
[ 336.299334] x15: 0000000000000000 x14: 0ffffffffffffffe
[ 336.304677] x13: 0000000000000030 x12: 0101010101010101
[ 336.310021] x11: 7f7f7f7f7f7f7f7f x10: 0000000000000860
[ 336.315362] x9 : ffffffc07d6d78b0 x8 : ffffffc0f62c08c0
[ 336.320704] x7 : 0000000000000001 x6 : 000000018b55dfa8
[ 336.326046] x5 : 0000000000000000 x4 : ffffffc0fb3221d0
[ 336.331387] x3 : 000000000000719c x2 : 0000000000000000
[ 336.336730] x1 : ffffff8000000021 x0 : ffffff8000000001
[ 336.342072]
[ 336.343576] Process v4l2-ctl (pid: 1094, stack limit = 0xffffffc07d6d4020)
[ 336.350456] Call trace:
[ 336.352916] [<ffffffc0007b6d14>] cil_write+0xc/0x14
[ 336.357806] [<ffffffc0007b47ac>] tegra_channel_vi_csi_recover+0x6c/0x180
[ 336.364515] [<ffffffc0007b48dc>] tegra_channel_ec_recover+0x1c/0x28
[ 336.370790] [<ffffffc0007b4ea4>] tegra_channel_capture_done+0x1a0/0x1fc
[ 336.377410] [<ffffffc0007b54c0>] vi2_channel_stop_streaming+0x48/0x128
[ 336.383947] [<ffffffc0007b0e38>] tegra_channel_stop_streaming+0x20/0x2c
[ 336.390581] [<ffffffc0007a7140>] __vb2_queue_cancel+0x34/0x138
[ 336.396423] [<ffffffc0007a729c>] vb2_core_streamoff+0x58/0xb0
[ 336.402180] [<ffffffc0007a97a8>] vb2_streamoff+0x48/0x50
[ 336.407504] [<ffffffc0007a9e68>] vb2_ioctl_streamoff+0x40/0x4c
[ 336.413352] [<ffffffc00078fdac>] v4l_streamoff+0x20/0x28
[ 336.418680] [<ffffffc000795804>] __video_do_ioctl+0x17c/0x298
[ 336.424434] [<ffffffc000795640>] video_usercopy+0x474/0x4a0
[ 336.430014] [<ffffffc000795680>] video_ioctl2+0x14/0x1c
[ 336.435249] [<ffffffc00078eb6c>] v4l2_ioctl+0xc8/0xe4
[ 336.440332] [<ffffffc0001d3140>] vfs_ioctl+0x18/0x40
[ 336.445309] [<ffffffc0001d3e0c>] do_vfs_ioctl+0x1f4/0x208
[ 336.450716] [<ffffffc0001d3e7c>] SyS_ioctl+0x5c/0x8c
[ 336.455702] [<ffffffc0000847f0>] el0_svc_naked+0x24/0x28
[ 336.461026] ---[ end trace db719a1075785bc5 ]---
olhx
July 20, 2018, 10:06am
10
To follow on the previous post #9 , we see no trace:
# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 0/0 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
Didn’t aware it’s TX1 only the TX2 have trace information.
Please enable the vi/csi log by below command.
echo file csi2_fops.c +p > /sys/kernel/debug/dynamic_debug/control
echo file vi2_fops.c +p > /sys/kernel/debug/dynamic_debug/control
Hi ShaneCCC,
Here’s the output with the traces:
[ 259.368686] video4linux video0: Syncpoint already enabled at capture done!0
[ 259.454544] video4linux video0: MW_ACK_DONE syncpoint time out!0
[ 259.460658] video4linux video0: TEGRA_VI_CSI_ERROR_STATUS 0x0000000a
[ 259.467080] vi 54080000.vi: TEGRA_CSI_PIXEL_PARSER_STATUS 0x00000000
[ 259.473495] vi 54080000.vi: TEGRA_CSI_CIL_STATUS 0x00000000
[ 259.479111] vi 54080000.vi: TEGRA_CSI_CILX_STATUS 0x00000000
and rest of the error:
[ 259.484847] Unable to handle kernel NULL pointer dereference at virtual address 00000101
[ 259.492981] pgd = ffffffc071e56000
[ 259.496414] [00000101] *pgd=00000000fb4e7003, *pud=00000000fb4e7003, *pmd=0000000000000000
[ 259.504813] Internal error: Oops: 96000046 [#1] PREEMPT SMP
[ 259.510398] Modules linked in: bcmdhd bluedroid_pm [last unloaded: fpga_mgr]
[ 259.517530] CPU: 2 PID: 1091 Comm: v4l2-ctl Tainted: G W 4.4.38-qcam5-g874c072-dirty #4
[ 259.526675] Hardware name: jetson_tx1 (DT)
[ 259.530781] task: ffffffc079c2b200 ti: ffffffc071e5c000 task.ti: ffffffc071e5c000
[ 259.538317] PC is at tpg_write+0xc/0x14
[ 259.542171] LR is at tegra_csi_error_recover+0xa8/0x1e4
[ 259.547404] pc : [<ffffffc0007b6a74>] lr : [<ffffffc0007b7488>] pstate: 60000145
[ 259.554811] sp : ffffffc071e5fa10
[ 259.558133] x29: ffffffc071e5fa10 x28: 0000000000000068
[ 259.563480] x27: 0000000000000014 x26: ffffffc07d6d4598
[ 259.568825] x25: 0000000000000214 x24: 0000000000000000
[ 259.574170] x23: 0000000000000020 x22: 0000000000000001
[ 259.579513] x21: ffffffc07d6d4600 x20: 0000000000000000
[ 259.584856] x19: ffffffc0f904da18 x18: 0000000000000000
[ 259.590200] x17: 0000000000000000 x16: 0000000000000000
[ 259.595541] x15: 0000000000000000 x14: 0ffffffffffffffe
[ 259.600883] x13: 0000000000000018 x12: 0101010101010101
[ 259.606226] x11: ffffffff7f7f7f7f x10: fefefefdff68752d
[ 259.611570] x9 : 7f7f7f7f7f7f7f7f x8 : ffffffc0f904da18
[ 259.616912] x7 : 0000000000000002 x6 : ffffffc0f96d71e8
[ 259.622256] x5 : ffffffc0f96d71e8 x4 : ffffffc07d503020
[ 259.627599] x3 : ffffffc0f904da28 x2 : 0000000000000001
[ 259.632942] x1 : 0000000000000101 x0 : 0000000000000101
[ 259.638283]
[ 259.639786] Process v4l2-ctl (pid: 1091, stack limit = 0xffffffc071e5c020)
[ 259.646667] Call trace:
[ 259.649129] [<ffffffc0007b6a74>] tpg_write+0xc/0x14
[ 259.654024] [<ffffffc0007b44e4>] tegra_channel_vi_csi_recover+0x6c/0x180
[ 259.660732] [<ffffffc0007b4614>] tegra_channel_ec_recover+0x1c/0x28
[ 259.667008] [<ffffffc0007b4bdc>] tegra_channel_capture_done+0x1a0/0x1fc
[ 259.673632] [<ffffffc0007b51f8>] vi2_channel_stop_streaming+0x48/0x128
[ 259.680172] [<ffffffc0007b0b70>] tegra_channel_stop_streaming+0x20/0x2c
[ 259.686804] [<ffffffc0007a6e78>] __vb2_queue_cancel+0x34/0x138
[ 259.692647] [<ffffffc0007a6fd4>] vb2_core_streamoff+0x58/0xb0
[ 259.698404] [<ffffffc0007a94e0>] vb2_streamoff+0x48/0x50
[ 259.703725] [<ffffffc0007a9ba0>] vb2_ioctl_streamoff+0x40/0x4c
[ 259.709575] [<ffffffc00078fae4>] v4l_streamoff+0x20/0x28
[ 259.714903] [<ffffffc00079553c>] __video_do_ioctl+0x17c/0x298
[ 259.720656] [<ffffffc000795378>] video_usercopy+0x474/0x4a0
[ 259.726238] [<ffffffc0007953b8>] video_ioctl2+0x14/0x1c
[ 259.731472] [<ffffffc00078e8a4>] v4l2_ioctl+0xc8/0xe4
[ 259.736551] [<ffffffc0001d3140>] vfs_ioctl+0x18/0x40
[ 259.741529] [<ffffffc0001d3e0c>] do_vfs_ioctl+0x1f4/0x208
[ 259.746937] [<ffffffc0001d3e7c>] SyS_ioctl+0x5c/0x8c
[ 259.751928] [<ffffffc0000847f0>] el0_svc_naked+0x24/0x28
[ 259.757252] ---[ end trace e586cbcece407496 ]---
Looks like the output resolution didn’t as expect.
External Media
Thanks ShaneCCC for pointing that out.
We’re still getting the same error.
We’re working with Leopard Imaging, but we’re having difficulties to know the exact value for line_length. We’re working with imx264 sensor.
When you say v4l2-ctl command must succeed, in fact it works as Leopard Imaging confirm us:
root@tegra-ubuntu:/home/nvidia# cat GET_2MP_V4L2_RAW
v4l2-ctl -d /dev/video0 -w --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=50 --stream-to=mydata.raw
root@tegra-ubuntu:/home/nvidia# ./GET_2MP_V4L2_RAW
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
Width/Height : 1920/1080
Pixel Format : 'RG12'
Field : None
Bytes per Line : 3840
Size Image : 4147200
Colorspace : sRGB
Transfer Function : Default
YCbCr Encoding : Default
Quantization : Default
Flags :
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
Index : 0
Type : Video Capture
Flags : mapped, done
Field : None
Sequence : 0
Length : 4147200
Bytesused: 4147200
Timestamp: 25673.477127s (Monotonic, End-of-Frame)
Index : 1
Type : Video Capture
Flags : mapped, done
Field : None
Sequence : 1
Length : 4147200
Bytesused: 4147200
Timestamp: 25673.510441s (Monotonic, End-of-Frame)
Index : 2
Type : Video Capture
Flags : mapped, done
Field : None
Sequence : 2
Length : 4147200
Bytesused: 4147200
Timestamp: 25673.543758s (Monotonic, End-of-Frame)
Index : 3
Type : Video Capture
Flags : mapped, done
Field : None
Sequence : 3
Length : 4147200
Bytesused: 4147200
Timestamp: 25673.577273s (Monotonic, End-of-Frame)
Index : 0
Type : Video Capture
Flags : mapped, done
Field : None
Sequence : 4
Length : 4147200
Bytesused: 4147200
Timestamp: 25673.710696s (Monotonic, End-of-Frame)
...
Index : 0
Type : Video Capture
Flags : mapped, done
Field : None
Sequence : 48
Length : 4147200
Bytesused: 4147200
Timestamp: 25679.577081s (Monotonic, End-of-Frame)
Index : 1
Type : Video Capture
Flags : mapped, done
Field : None
Sequence : 49
Length : 4147200
Bytesused: 4147200
Timestamp: 25679.710464s (Monotonic, End-of-Frame)
6.95 fps
VIDIOC_STREAMOFF: ok
root@tegra-ubuntu:/home/nvidia#
But when we slice the raw file to get individual frames, we find black frames between correct ones.
How is this related to our original EGL error?
Thanks
Beside our Gstreamer chain failing, nvgstcapture-1.0 fails also with nvcamera-daemon crashing.
Leopard Imaging suggests nvidia’s team to lookup our dts which I have just attached.
tegra210-icar-qcam5-2161-0.dts.log (590 KB)
The resolution in DT is 1936x1080 but the kernel driver report is 1920x1080, why?
Format Video Capture:
Width/Height : 1920/1080
Pixel Format : 'RG12'
active_w = "1936";
active_h = "1080";
Also remove the pixel_t from the DT, pixel_t is replace by below.
dynamic_pixel_bit_depth = "12";
csi_pixel_bit_depth = "12";
pixel_phase = "grbg";
Resolution in DT is referencing active pixels which are: 1936 x 1096
Driver is referencing recording pixels which are: 1920 x 1080
I’ve removed pixel_t.
Can you confirm line_length and pix_clk_hz?
I don’t get it. No matter what they should be the same. For the line_length and pix_clk_hz you need to get it from sensor vendor. It’s sensor depend.