Is it Possible Add timestamp to Camera sensor raw data?

For our project, We use two camera device with one external trigger source.

In order to have the app can get the camera raw frame data at the same time.

Is it Possible Add timestamp to Camera sensor raw data on the firsts 4/8 bytes?

hello 27653276,

please check couple of questions as below.

  1. may I know which JetPack release you’re working with.
  2. please refer to Camera Architecture Stack, may I know what’s your use-case, and which approach you would like to access your camera sensor.

Hi JerryChang,

We are using V4L2 Application to get camera raw Image data base on Jetpack 3.3.
We use the same PWM trigger for the left and right camera.
Currently,I can get the two sensor rawdata by V4L2, The APP want to confirm if the two frame raw data is come from the same time point.
So I think if we can add timestamp to Head of sensor rawd data when CSI Module capture the data.

Thanks,

hello 27653276,

we had kernel patch to fix VI4 timestamp issues, please refer to Topic 1020202 for the patches.

in addition,
since there’s vb2 buffer to gather timestamps, do you still need to add that into the head of sensor raw?
thanks

Hi JerryChang,

Thank you very much for your support.
Just a confirm, For the patch, the timestamp will been added the head of sensor raw??
What’s size for the timestamps? 4 bytes or 8 bytes?

Can you let me know what's vb2 buffer? that's another soluation to get timestamp?

if so, that's more better way to get timestamp, I think We will need it. can you send us the detail information?

Thanks,
Martin

hello 27653276,

there’s an alternative way to have timestamp for each capture frames,
vb2 means the video buffer information for v4l2, this also the data struct for v4l2 applications.
please refer to below for vb2 buffer.

struct vb2_v4l2_buffer {
  	struct vb2_buffer	vb2_buf;
  
  	__u32			flags;
  	__u32			field;
  	struct timeval		timestamp;
  	struct v4l2_timecode	timecode;
  	__u32			sequence;
};

for your development based-on JetPack-3.3, you should apply below two kernel patches for VI driver.

  1. please check Topic 1038067 for the patches to fix FE syncpt wait.
  2. please check Topic 1020202 to fix missing timestamp for vi4.

you may also refer to below kernel sources for more details.
thanks
$TOP/kernel_src/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/channel.c
$TOP/kernel_src/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/vi4_fops.c

Hi JerryChang,

In the Topic 1038067(Aug6_Topic1038067.tar.gz), there are two patch file:
0001-drivers-camera-fix-FE-syncpt-wait.patch
0001-media-tegra-fix-missing-timestamp-for-vi4.patch

It look like that the senond patch file(0001-media-tegra-fix-missing-timestamp-for-vi4.patch
) is the same as the patch in the Topic 1020202.

So Just need to apply the Topic 1038067, Is it correctly?

that’s correct, please apply below two patches.
0001-drivers-camera-fix-FE-syncpt-wait.patch
0001-media-tegra-fix-missing-timestamp-for-vi4.patch

Hi JerryChang,

After apply above two patch,
With following code to get timestamp from head of sensor raw data.
But it seem like that it is not working

{
struct timeval timestamp;
memcpy(&timestamp,image_buffer,sizeof(struct timeval));
printf(“timestamp.tv_sec = %ld timestamp.tv_usec= %ld\n”,timestamp.tv_sec,timestamp.tv_usec);
}
image_buffer is the buffer for sensor raw data.
the log message is as following:
timestamp.tv_sec = 288516322261795841 timestamp.tv_usec= 311034526549672993
timestamp.tv_sec = 302027189860238353 timestamp.tv_usec= 311034526552884241
timestamp.tv_sec = 288516180523680865 timestamp.tv_usec= 306530858207085552
timestamp.tv_sec = 306530858207085569 timestamp.tv_usec= 306530995651281937
timestamp.tv_sec = 288516459696555041 timestamp.tv_usec= 283731522533524513
timestamp.tv_sec = 302027258578666497 timestamp.tv_usec= 302027396016571441
timestamp.tv_sec = 297523311058945057 timestamp.tv_usec= 288516111805252625
timestamp.tv_sec = 306530720768132097 timestamp.tv_usec= 302027052417025057
timestamp.tv_sec = 311034663994983457 timestamp.tv_usec= 311034526553932833
timestamp.tv_sec = 288516390974915585 timestamp.tv_usec= 293020059322876912
timestamp.tv_sec = 288516459699700737 timestamp.tv_usec= 306530926927610929
timestamp.tv_sec = 297523521515488224 timestamp.tv_usec= 302027258576503889
timestamp.tv_sec = 288516528416031793 timestamp.tv_usec= 293019990609691697
timestamp.tv_sec = 306530579033162769 timestamp.tv_usec= 297523727665464369
timestamp.tv_sec = 306530858209182785 timestamp.tv_usec= 297523521510179889
timestamp.tv_sec = 297523727671821377 timestamp.tv_usec= 315538126181303361
timestamp.tv_sec = 320041588369720369 timestamp.tv_usec= 302027121140761617
timestamp.tv_sec = 302027052419187729 timestamp.tv_usec= 297523452797060129
timestamp.tv_sec = 288516459697603617 timestamp.tv_usec= 302026979405792224
timestamp.tv_sec = 311034663996032065 timestamp.tv_usec= 302027327297094689

Thanks,

Currently, We just use simple v4l2 buffer to get sensor raw data.
I must use vb2 buffer to get timestamp and sensor raw data?
if so, Have any sample to use vb2 buffer to get timestamp?

hello 27653276,

please refer to below kernel source,
we fill-in the timestamp with the capture time of start-of-frame.
thanks

<i>vi4_fops.c:</i>
static bool vi_notify_wait(struct tegra_channel *chan,
                struct tegra_channel_buffer *buf,
                struct timespec *ts)
{
...
                        *ts = ns_to_timespec((s64)status.sof_ts);

<i>channel.c:</i>
void set_timestamp(struct tegra_channel_buffer *buf,
                        const struct timespec *ts)
{
        buf->buf.timestamp.tv_sec = ts->tv_sec;
        buf->buf.timestamp.tv_usec = ts->tv_nsec / NSEC_PER_USEC;
}

I confirmed that We are using the code have the related updated.
You means that the code apply two above patch should work fine with add timestamp to start-of-frame?

Hello JerryChang,

For your kernel source, the timestamp been added to timestamp of struct vb2_v4l2_buffer,

Can you confirm the timestamp will been copy to start-of-frame?

For our v4l2 application, use VIDIOC_DQBUF get the sensor raw data.

Thanks,

Hi,

You can extract the timestamp from the buffer obtained with the VIDIOC_QBUF ioctl as in the code below (where the obtained buffer is struct v4l2_buffer *buf). This is the same timestamp set in vi4_fops.c/channel.c previously.

struct timeval ts; 
ts = buf->timestamp;

Then you could add this information to the image data if that’s your requirement.

Hello jcaballero,

Thank you for your information.
I have got it working fine.

Thanks,