Hi,
We have a custom and non-standard image format from an image sensor. The format outputs 2 channels with 12 bits on the first and 8 bits on the 2nd channel. Is it possible for the TX2 hardware to receive this and use v4l2 for streaming? What parameters do we need to set in the sensor driver to enable this?
Processing doesn’t need to be done on the TX2 itself. We just need to dump the image data from the memory buffer and save it as a raw file.
Thanks.
hello 3dimi,
please note that, raw data sent to memory is represent by a 16-bit buffer.
there might be MSBs replicated depends on your input data, suggest you refer to Tegra X2 (Parker Series SoC) Technical Reference Manual → [VIDEO INPUT (VI)]-> [RAW Memory Formats] for details.
thanks
Hi Jerry,
Thanks for your guide.
I checked the details in the TRM and I started creating a sensor driver based on the RAW10 format. I used bayer type with rggb pixel phase and 10 bit in the device tree. I also used active width that is twice the actual width of the image from the sensor since it outputs 20 bpp.
However, I keep getting a CHANSEL_SHORT_FRAME error from the trace.
# tracer: nop
#
# entries-in-buffer/entries-written: 86/86 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
kworker/0:2-223 [000] ...1 58.862467: rtos_queue_peek_from_isr_failed: tstamp:2194533130 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 58.862471: rtcpu_start: tstamp:2194536508
kworker/0:2-223 [000] ...1 59.018496: rtos_queue_peek_from_isr_failed: tstamp:2199534320 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 59.174503: rtos_queue_peek_from_isr_failed: tstamp:2204534829 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 59.330655: rtos_queue_peek_from_isr_failed: tstamp:2209535664 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 59.486536: rtos_queue_peek_from_isr_failed: tstamp:2214536037 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 59.642553: rtos_queue_peek_from_isr_failed: tstamp:2219536517 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 59.798604: rtos_queue_peek_from_isr_failed: tstamp:2224537025 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 59.902564: rtcpu_vinotify_handle_msg: tstamp:2227988260 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2227987387 data:0x00000000
kworker/0:2-223 [000] ...1 59.954529: rtcpu_vinotify_handle_msg: tstamp:2229050837 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:2229050017 data:0x00000001
kworker/0:2-223 [000] ...1 59.954538: rtcpu_vinotify_handle_msg: tstamp:2229051420 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2229050020 data:0x00000000
kworker/0:2-223 [000] ...1 59.954545: rtos_queue_peek_from_isr_failed: tstamp:2229537537 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.110506: rtos_queue_peek_from_isr_failed: tstamp:2234538029 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.266533: rtos_queue_peek_from_isr_failed: tstamp:2239538423 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.266543: rtos_queue_peek_from_isr_failed: tstamp:2239802609 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.266545: rtos_queue_peek_from_isr_failed: tstamp:2239884610 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.266549: rtcpu_start: tstamp:2239886852
kworker/0:2-223 [000] ...1 60.318586: rtcpu_vinotify_handle_msg: tstamp:2240503348 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:2240502827 data:0x00000001
kworker/0:2-223 [000] ...1 60.370552: rtcpu_vinotify_handle_msg: tstamp:2241754978 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2241754377 data:0x00000000
kworker/0:2-223 [000] ...1 60.370561: rtcpu_vinotify_handle_msg: tstamp:2242817697 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:2242817010 data:0x00000001
kworker/0:2-223 [000] ...1 60.370564: rtcpu_vinotify_handle_msg: tstamp:2242818005 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2242817013 data:0x00000000
kworker/0:2-223 [000] ...1 60.422534: rtos_queue_peek_from_isr_failed: tstamp:2244885545 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.630483: rtos_queue_peek_from_isr_failed: tstamp:2249886049 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.786517: rtos_queue_peek_from_isr_failed: tstamp:2254886557 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 60.942478: rtos_queue_peek_from_isr_failed: tstamp:2259887035 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.098601: rtos_queue_peek_from_isr_failed: tstamp:2264887609 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.254527: rtos_queue_peek_from_isr_failed: tstamp:2269888123 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.308378: rtos_queue_peek_from_isr_failed: tstamp:2271553186 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.308395: rtos_queue_peek_from_isr_failed: tstamp:2271645315 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.308403: rtcpu_start: tstamp:2271647895
kworker/0:2-223 [000] ...1 61.308411: rtcpu_vinotify_handle_msg: tstamp:2271791182 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:2271790649 data:0x000000a0
kworker/0:2-223 [000] ...1 61.308415: rtcpu_vinotify_handle_msg: tstamp:2271791982 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2271791461 data:0x00000000
kworker/0:2-223 [000] ...1 61.358582: rtcpu_vinotify_handle_msg: tstamp:2272854784 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:2272854096 data:0x00000001
kworker/0:2-223 [000] ...1 61.358597: rtcpu_vinotify_handle_msg: tstamp:2272855089 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2272854099 data:0x00000000
kworker/0:2-223 [000] ...1 61.462524: rtos_queue_peek_from_isr_failed: tstamp:2276646398 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.618553: rtos_queue_peek_from_isr_failed: tstamp:2281646824 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.774544: rtos_queue_peek_from_isr_failed: tstamp:2286647345 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 61.930523: rtos_queue_peek_from_isr_failed: tstamp:2291647839 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.086524: rtos_queue_peek_from_isr_failed: tstamp:2296648344 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.242579: rtos_queue_peek_from_isr_failed: tstamp:2301648888 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.298542: rtos_queue_peek_from_isr_failed: tstamp:2303429060 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.350514: rtos_queue_peek_from_isr_failed: tstamp:2303596080 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.350525: rtcpu_start: tstamp:2303598805
kworker/0:2-223 [000] ...1 62.350530: rtcpu_vinotify_handle_msg: tstamp:2304332196 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2304331656 data:0x00000000
kworker/0:2-223 [000] ...1 62.402531: rtcpu_vinotify_handle_msg: tstamp:2305394931 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:2305394289 data:0x00000001
kworker/0:2-223 [000] ...1 62.402540: rtcpu_vinotify_handle_msg: tstamp:2305395136 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2305394292 data:0x00000000
kworker/0:2-223 [000] ...1 62.506510: rtos_queue_peek_from_isr_failed: tstamp:2308597209 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.662523: rtos_queue_peek_from_isr_failed: tstamp:2313597714 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.818570: rtos_queue_peek_from_isr_failed: tstamp:2318598155 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 62.974525: rtos_queue_peek_from_isr_failed: tstamp:2323598862 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 63.130606: rtos_queue_peek_from_isr_failed: tstamp:2328599360 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 63.286520: rtos_queue_peek_from_isr_failed: tstamp:2333599864 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 63.338544: rtos_queue_peek_from_isr_failed: tstamp:2335360075 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 63.338554: rtos_queue_peek_from_isr_failed: tstamp:2335948964 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 63.338558: rtcpu_start: tstamp:2335952993
kworker/0:2-223 [000] ...1 63.390514: rtcpu_vinotify_handle_msg: tstamp:2336872482 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2336871840 data:0x00000000
kworker/0:2-223 [000] ...1 63.442545: rtcpu_vinotify_handle_msg: tstamp:2337935268 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:2337934475 data:0x00000001
kworker/0:2-223 [000] ...1 63.442561: rtcpu_vinotify_handle_msg: tstamp:2337936016 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2337934478 data:0x00000000
kworker/0:2-223 [000] ...1 63.546591: rtos_queue_peek_from_isr_failed: tstamp:2340950068 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 63.702555: rtos_queue_peek_from_isr_failed: tstamp:2345950572 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 63.858596: rtos_queue_peek_from_isr_failed: tstamp:2350951081 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.014530: rtos_queue_peek_from_isr_failed: tstamp:2355951629 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.170599: rtos_queue_peek_from_isr_failed: tstamp:2360952342 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.326503: rtos_queue_peek_from_isr_failed: tstamp:2365952611 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.378528: rtos_queue_peek_from_isr_failed: tstamp:2367686308 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.378553: rtos_queue_peek_from_isr_failed: tstamp:2368325097 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.378563: rtcpu_start: tstamp:2368328091
kworker/0:2-223 [000] ...1 64.430537: rtcpu_vinotify_handle_msg: tstamp:2369412673 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2369412038 data:0x00000000
kworker/0:2-223 [000] ...1 64.482609: rtcpu_vinotify_handle_msg: tstamp:2370475463 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:2370474670 data:0x00000001
kworker/0:2-223 [000] ...1 64.482622: rtcpu_vinotify_handle_msg: tstamp:2370475998 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2370474673 data:0x00000000
kworker/0:2-223 [000] ...1 64.534628: rtos_queue_peek_from_isr_failed: tstamp:2373326198 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.742637: rtos_queue_peek_from_isr_failed: tstamp:2378326723 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 64.898544: rtos_queue_peek_from_isr_failed: tstamp:2383327432 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 65.054513: rtos_queue_peek_from_isr_failed: tstamp:2388327534 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 65.210703: rtos_queue_peek_from_isr_failed: tstamp:2393328224 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 65.366583: rtos_queue_peek_from_isr_failed: tstamp:2398328735 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 65.470706: rtcpu_vinotify_handle_msg: tstamp:2401952958 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2401952229 data:0x00000000
kworker/0:2-223 [000] ...1 65.522476: rtcpu_vinotify_handle_msg: tstamp:2403015684 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:2403014863 data:0x00000001
kworker/0:2-223 [000] ...1 65.522482: rtcpu_vinotify_handle_msg: tstamp:2403016200 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2403014866 data:0x00000000
kworker/0:2-223 [000] ...1 65.522487: rtos_queue_peek_from_isr_failed: tstamp:2403329052 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 65.678487: rtos_queue_peek_from_isr_failed: tstamp:2408329573 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 65.834556: rtos_queue_peek_from_isr_failed: tstamp:2413330329 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 65.990523: rtos_queue_peek_from_isr_failed: tstamp:2418330759 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 66.146679: rtos_queue_peek_from_isr_failed: tstamp:2423331275 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 66.302526: rtos_queue_peek_from_isr_failed: tstamp:2428331772 queue:0x0b4a3c58
kworker/0:2-223 [000] ...1 66.458543: rtos_queue_peek_from_isr_failed: tstamp:2431982924 queue:0x0b4a3c58
I already tried changing the active height. I had set active_h = 1 and it still shows this short frame error. Do you have any idea what could be the problem?
hello 3dimi,
VI hardware engine parse the configuration from device tree properties settings, to describe one complete sensor output, please consider horizontal timing size as LINE, vertical timing as FRAME.
hence, the failure of CHANSEL_SHORT_FRAME means the vertical direction (active_h) of signaling is “less” than VI’s expectation.
may I know what’s your sensor actual output size?
thanks
Hi JerryChang,
The output of the sensor is 1920x1280 with 1 embedded data line. I already used these values but still getting the short frame error.
hello 3dimi,
please also check sensor specification, check the sensor diagram for effective margins.
suggest you also refer to Sensor Driver Programming Guide and review your sensor device tree.
thanks
Hi JerryChang,
The sensor effective margins are for the line and frame blanking right?
I already counterchecked the sensor settings with the spec and from the supplier. It matches my setting and it also matches the registers from the serializer-deserializer setup that we are using.
The problem is that even with very large or very low values of active_h in the device tree and kernel driver, the error stays the same (short frame error).
hello 3dimi,
please also check the deserializer hardware output signaling is match your configuration.
this output signaling should also contain FS/FE/FS…FE packet.
BTW, since you’re working with serializer-deserializer setup.
is there a capability to enable test-pattern-generator from the device?
thanks
Hi JerryChang,
We did try TPG from the serdes setup. However, the TPG format is RGB888, which is different from the sensor output. We were able to get good stream of the test pattern.
hello 3dimi,
may I know which JetPack release you’re working with.
could you please also share your commands.
according to Camera Architecture Stack, there are two ways to access camera sensor.
please have these two ways to access your sensor streaming to check if any difference,
for example.
v4l2-ctl
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100
and
gst-launch
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvoverlaysink -ev
Hi JerryChang,
We are still using JetPack 3.3.
We currently use the qv4l2 interface in checking the stream. I also tried the v4l2-ctl example that you provided but still getting the same error in the trace.
hello 3dimi,
back to your 1st comment,
The format outputs 2 channels with 12 bits on the first and 8 bits on the 2nd channel.
may I know how your serializer-deserializer driver handle this,
is it continuous outputting 1920x1080/10-bit streaming to tegra-csi?
thanks
Hi JerryChang,
The serializer-deserializer accurately reports the line size in bytes (4800 bytes). We are using TI953-960 serdes pair. The number of lines is also accurate. The total number of lines is actually 1280 + 1 embedded line.
There are no reported CSI errors in the registers of the serdes.
hello 3dimi,
could you please attach sensor device tree here for us checking.
thanks
Hi JerryChang,
Here is the mode section of the device tree:
mode0 {
mclk_khz = "25000";
num_lanes = "4";
tegra_sinterface = "serial_a";
discontinuous_clk = "yes";
dpcm_enable = "false";
cil_settletime = "0";
dynamic_pixel_bit_depth = "10";
csi_pixel_bit_depth = "10";
mode_type = "bayer";
pixel_phase = "rggb";
active_w = "3840";
active_h = "1280";
readout_orientation = "0";
line_length = "5000";
inherent_gain = "1";
mclk_multiplier = "10";
pix_clk_hz = "100000000";
min_gain_val = "0"; /* dB */
max_gain_val = "30"; /* dB */
min_hdr_ratio = "1";
max_hdr_ratio = "16";
min_framerate = "25";
max_framerate = "25";
min_exp_time = "118";
max_exp_time = "660000";
embedded_metadata_height = "1";
};
Is this what you need?
Hi JerryChang,
Will the CSI-VI blocks of the TX2 receive the incoming data if the data ID in the packet header is different from the format that is set in the sensor driver and device tree?
The data packet header for the sensor stream is different from the RAW10 data type ID.
hello 3dimi,
yes, here are my questions and suggestions,
may I have your comments about why you configure this double of your 1920x1080 sensor streaming?
active_w = "3840";
please check our R32.1 Sensor Driver Programming Guide → [Sensor Pixel Clock] session for the calculation formula.
pix_clk_hz = "100000000";
this were used for WDR sensors, please keep min_hdr_ratio=max_hdr_ratio for avoiding calculation failure.
min_hdr_ratio = "1";
max_hdr_ratio = "16";
hello 3dimi,
mostly common issues we had were properties settings mismatch the actual streaming.
actually, VI block use the configurations from sensor device tree to receive the data streaming.
suggest you co-work with your hardware engineer to probe the signal and make sure it follow the MIPI spec.
thanks
Hi JerryChang,
I set the active_w as twice the actual because I am using RAW10 but the actual bits per pixel of the sensor is 20 bpp.
I also tried several settings for pix_clk_hz and it still doesn’t work.
Hi JerryChang,
The same serdes link and sensor have been streaming successfully in a different platform. We think there is no issue with the MIPI signal going into the TX2.
We will double check again whether there is an issue.