Delay in Visualization of Camera and LiDAR Data

j_mueller · October 28, 2024, 9:50am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Jetson Orin AGX
• DeepStream Version: 7.0
• JetPack Version (valid for Jetson only): 6.0
• TensorRT Version: 8.6.2.3
• Issue Type( questions, new requirements, bugs): Question

Hi there,

I want to visualize camera and LiDAR data simultaneously in Deepstream. For the camera stream, I’m using Aravis, and for the LiDAR, I’ve written a plugin myself. When I display both streams with NvDs3D_CreateGlesEnsembleRender, the camera stream is shown in “real-time”, but the LiDAR stream has a delay of roughly 1.5 seconds, even though the timestamps from a buffer probe are identical within about 40ms.

If I display only the LiDAR data without the camera stream, there is no delay.

My pipline is:

gstparsebin (aravis) → 3dVideoDatabridge → 3dMultiSensorMixer.0
LiDARSourcePlugIn → 3dMultiSensorMixer.1

3dMultiSensorMixer → 3dGlesEnsembleRender

What could be causing this?

yuweiw · October 29, 2024, 2:37am

Could you confirm that the lidar timestamp is aligned with the video timestamp?
Also could you attach your config file?

j_mueller · October 29, 2024, 6:33am

Thank you for the response! LiDAR and camera data match within approximately 40ms after the pipeline buildup phase (After 240ms) . Here is a snippet of the probes:

There is a src pad probe for LiDAR TimeStamp (Probe at Dataloader), Camera TimeStamp (Probe at Videobridge) and Mixer TimeStamp (Probe at Mixer).

For Renderer its a sink pad probe at Datarenderer

Lidar TimeStamp: 0
Lidar TimeStamp: 40
Lidar TimeStamp: 80
Play...
Camera TimeStamp: 80
Mixer TimeStamp: 0
Mixer TimeStamp: 40
Renderer TimeStamp: 0
Mixer TimeStamp: 80
Renderer TimeStamp: 40
Renderer TimeStamp: 80
Camera TimeStamp: 120
Camera TimeStamp: 160
Lidar TimeStamp: 120
Mixer TimeStamp: 120
Renderer TimeStamp: 120
Camera TimeStamp: 200
Lidar TimeStamp: 160
Mixer TimeStamp: 160
Renderer TimeStamp: 160
Lidar TimeStamp: 200
Mixer TimeStamp: 200
Renderer TimeStamp: 200
Camera TimeStamp: 240
Lidar TimeStamp: 240
Mixer TimeStamp: 240
Renderer TimeStamp: 240
Camera TimeStamp: 280
Lidar TimeStamp: 280
Mixer TimeStamp: 280
Renderer TimeStamp: 280
Camera TimeStamp: 320
Lidar TimeStamp: 320
Mixer TimeStamp: 320
Camera TimeStamp: 360
Renderer TimeStamp: 320
Lidar TimeStamp: 360
Mixer TimeStamp: 360
Renderer TimeStamp: 360
Camera TimeStamp: 400
Lidar TimeStamp: 400
Mixer TimeStamp: 400
Renderer TimeStamp: 400
Camera TimeStamp: 440
Lidar TimeStamp: 440
Mixer TimeStamp: 440
Renderer TimeStamp: 440
Camera TimeStamp: 480
Lidar TimeStamp: 480
Mixer TimeStamp: 480
Renderer TimeStamp: 480
Camera TimeStamp: 520
Lidar TimeStamp: 520
Mixer TimeStamp: 520
Renderer TimeStamp: 520
Camera TimeStamp: 560
Lidar TimeStamp: 560
Mixer TimeStamp: 560
Renderer TimeStamp: 560
Camera TimeStamp: 600
Lidar TimeStamp: 600
Mixer TimeStamp: 600
Renderer TimeStamp: 600
Camera TimeStamp: 640
Lidar TimeStamp: 640
Mixer TimeStamp: 640
Renderer TimeStamp: 640
Camera TimeStamp: 680
Lidar TimeStamp: 680
Mixer TimeStamp: 680
Renderer TimeStamp: 680
Camera TimeStamp: 720
Lidar TimeStamp: 720
Mixer TimeStamp: 720
Renderer TimeStamp: 720
Camera TimeStamp: 760
Lidar TimeStamp: 760
Mixer TimeStamp: 760
Renderer TimeStamp: 760
Camera TimeStamp: 800
Lidar TimeStamp: 800
Mixer TimeStamp: 800
Renderer TimeStamp: 800
Camera TimeStamp: 840
Lidar TimeStamp: 840
Mixer TimeStamp: 840
Renderer TimeStamp: 840
Camera TimeStamp: 880
Lidar TimeStamp: 880
Mixer TimeStamp: 880
Renderer TimeStamp: 880
Camera TimeStamp: 920
Lidar TimeStamp: 920
Mixer TimeStamp: 920
Renderer TimeStamp: 920
Camera TimeStamp: 960
Lidar TimeStamp: 960
Mixer TimeStamp: 960
Renderer TimeStamp: 960
Camera TimeStamp: 1000
Lidar TimeStamp: 1000
Mixer TimeStamp: 1000
Renderer TimeStamp: 1000
Camera TimeStamp: 1040
Lidar TimeStamp: 1040
Mixer TimeStamp: 1040
Renderer TimeStamp: 1040
Camera TimeStamp: 1080
Lidar TimeStamp: 1080
Mixer TimeStamp: 1080
Renderer TimeStamp: 1080
Camera TimeStamp: 1120
Lidar TimeStamp: 1120
Mixer TimeStamp: 1120
Renderer TimeStamp: 1120
Camera TimeStamp: 1160
Lidar TimeStamp: 1160
Mixer TimeStamp: 1160
Renderer TimeStamp: 1160
Camera TimeStamp: 1200
Camera TimeStamp: 1240
Lidar TimeStamp: 1200
Mixer TimeStamp: 1200
Renderer TimeStamp: 1200
Lidar TimeStamp: 1240
Mixer TimeStamp: 1240
Renderer TimeStamp: 1240
Camera TimeStamp: 1280
Lidar TimeStamp: 1280
Mixer TimeStamp: 1280
Renderer TimeStamp: 1280
Camera TimeStamp: 1320
Lidar TimeStamp: 1320
Mixer TimeStamp: 1320
Renderer TimeStamp: 1320
Camera TimeStamp: 1360
Lidar TimeStamp: 1360
Mixer TimeStamp: 1360
Renderer TimeStamp: 1360
Camera TimeStamp: 1400
Lidar TimeStamp: 1400
Camera TimeStamp: 1440
Mixer TimeStamp: 1400
Renderer TimeStamp: 1400
Lidar TimeStamp: 1440
Mixer TimeStamp: 1440
Renderer TimeStamp: 1440
Camera TimeStamp: 1480
Lidar TimeStamp: 1480
Mixer TimeStamp: 1480
Renderer TimeStamp: 1480
Camera TimeStamp: 1520
Lidar TimeStamp: 1520
Mixer TimeStamp: 1520
Renderer TimeStamp: 1520
Camera TimeStamp: 1560
Lidar TimeStamp: 1560
Mixer TimeStamp: 1560
Renderer TimeStamp: 1560
Camera TimeStamp: 1600
Lidar TimeStamp: 1600
Mixer TimeStamp: 1600
Renderer TimeStamp: 1600
Camera TimeStamp: 1640
Lidar TimeStamp: 1640
Mixer TimeStamp: 1640
Renderer TimeStamp: 1640
Camera TimeStamp: 1680
Lidar TimeStamp: 1680
Mixer TimeStamp: 1680
Renderer TimeStamp: 1680
Camera TimeStamp: 1720
Lidar TimeStamp: 1720
Mixer TimeStamp: 1720
Renderer TimeStamp: 1720
Camera TimeStamp: 1760
Lidar TimeStamp: 1760
Mixer TimeStamp: 1760
Renderer TimeStamp: 1760
Lidar TimeStamp: 1800
Camera TimeStamp: 1800
Mixer TimeStamp: 1800
Renderer TimeStamp: 1800
Lidar TimeStamp: 1840
Lidar TimeStamp: 1880
Camera TimeStamp: 1840
Mixer TimeStamp: 1840
Renderer TimeStamp: 1840
Lidar TimeStamp: 1920
Camera TimeStamp: 1880
Mixer TimeStamp: 1880
Renderer TimeStamp: 1880
Lidar TimeStamp: 1960
Camera TimeStamp: 1920
Mixer TimeStamp: 1920
Renderer TimeStamp: 1920
Lidar TimeStamp: 2000
Camera TimeStamp: 1960
Mixer TimeStamp: 1960
Renderer TimeStamp: 1960
Lidar TimeStamp: 2040
Camera TimeStamp: 2000
Mixer TimeStamp: 2000
Renderer TimeStamp: 2000
Lidar TimeStamp: 2080
Camera TimeStamp: 2040
Mixer TimeStamp: 2040
Renderer TimeStamp: 2040
Lidar TimeStamp: 2120
Camera TimeStamp: 2080
Mixer TimeStamp: 2080
Renderer TimeStamp: 2080
Lidar TimeStamp: 2160
Camera TimeStamp: 2120
Mixer TimeStamp: 2120
Renderer TimeStamp: 2120
Lidar TimeStamp: 2200
Camera TimeStamp: 2160
Mixer TimeStamp: 2160
Renderer TimeStamp: 2160
Lidar TimeStamp: 2240
Camera TimeStamp: 2200
Mixer TimeStamp: 2200
Renderer TimeStamp: 2200
Lidar TimeStamp: 2280
Camera TimeStamp: 2240
Mixer TimeStamp: 2240
Renderer TimeStamp: 2240
Lidar TimeStamp: 2320
Camera TimeStamp: 2280
Mixer TimeStamp: 2280
Renderer TimeStamp: 2280
Lidar TimeStamp: 2360
Camera TimeStamp: 2320
Mixer TimeStamp: 2320
Renderer TimeStamp: 2320
Lidar TimeStamp: 2400
Camera TimeStamp: 2360
Mixer TimeStamp: 2360
Renderer TimeStamp: 2360
Lidar TimeStamp: 2440
Camera TimeStamp: 2400
Mixer TimeStamp: 2400
Renderer TimeStamp: 2400
Lidar TimeStamp: 2480
Camera TimeStamp: 2440
Mixer TimeStamp: 2440
Renderer TimeStamp: 2440
Lidar TimeStamp: 2520
Camera TimeStamp: 2480
Mixer TimeStamp: 2480
Renderer TimeStamp: 2480
Lidar TimeStamp: 2560
Camera TimeStamp: 2520
Mixer TimeStamp: 2520
Renderer TimeStamp: 2520
Lidar TimeStamp: 2600
Camera TimeStamp: 2560
Mixer TimeStamp: 2560
Renderer TimeStamp: 2560
Lidar TimeStamp: 2640
Camera TimeStamp: 2600
Mixer TimeStamp: 2600
Renderer TimeStamp: 2600
Lidar TimeStamp: 2680
Camera TimeStamp: 2640
Mixer TimeStamp: 2640
Renderer TimeStamp: 2640
Lidar TimeStamp: 2720
Camera TimeStamp: 2680
Mixer TimeStamp: 2680
Renderer TimeStamp: 2680
Lidar TimeStamp: 2760
Camera TimeStamp: 2720
Mixer TimeStamp: 2720
Renderer TimeStamp: 2720
Lidar TimeStamp: 2800
Camera TimeStamp: 2760
Mixer TimeStamp: 2760
Renderer TimeStamp: 2760
Lidar TimeStamp: 2840
Camera TimeStamp: 2800
Mixer TimeStamp: 2800
Renderer TimeStamp: 2800
Lidar TimeStamp: 2880
Camera TimeStamp: 2840
Mixer TimeStamp: 2840
Renderer TimeStamp: 2840
Lidar TimeStamp: 2920
Camera TimeStamp: 2880
Mixer TimeStamp: 2880
Renderer TimeStamp: 2880
Lidar TimeStamp: 2960
Camera TimeStamp: 2920
Mixer TimeStamp: 2920
Renderer TimeStamp: 2920
Lidar TimeStamp: 3000
Camera TimeStamp: 2960
Mixer TimeStamp: 2960
Renderer TimeStamp: 2960

And here is the config file:

name: video_source
type: ds3d::gstparsebin
link_to: videobridge_2d_to_3d
config_body:
  parse_bin: >-
    aravissrc camera-name=192.168.57.36 ! video/x-raw, format=RGB, framerate=25/1  ! nvvideoconvert nvbuf-memory-type=2 compute-hw=1 ! video/x-raw(memory:NVMM), format=RGBA ! queue 

# bridge (2d->3d)
---
name: videobridge_2d_to_3d
type: ds3d::databridge
link_to: multisensor_mixer_lidar_and_image.sink_0
in_caps: video/x-raw(memory:NVMM)
out_caps: ds3d/datamap
with_queue: src
custom_lib_path: libnvds_3d_video_databridge.so
custom_create_function: createVideoBridge2d3d
config_body:
  surface_to_image: True
  # wrap nvbufsurface into color frame of ds3d::Frame2DGuard
  output_image_key: DS3D::ColorFrame
  output_image_preproess_key: DS3D::VideoPreprocessTensor

# lidar source
---
name: lidarsource
type: ds3d::dataloader
link_to: multisensor_mixer_lidar_and_image.sink_1
out_caps: ds3d/datamap
custom_lib_path: libnvds_lidarsensorread.so
custom_create_function: createLidarSensorReader
config_body:
  source_id: 0
  #path of the frames, data file needs to be placed in order in this path
  points_num: 715 #points num
  fixed_points_num: False
  lidar_datatype: FP32 #FP32 FP16 INT8 INT32, only support FP32 this version
  mem_type: gpu #cpu gpu, only support cpu this version
  mem_pool_size: 4
  gpu_id: 0
  element_size: 4
  element_stride: 4
  output_datamap_key: DS3D::LidarXYZI_0

# mixer
---
name: multisensor_mixer_lidar_and_image
type: ds3d::datamixer
link_to: ds3d_sensor_fusion_render
in_caps: ds3d/datamap
out_caps: ds3d/datamap
#with_queue: src
custom_lib_path: libnvds_3d_multisensor_mixer.so
custom_create_function: createMultiSensorMixer
config_body:
  #timeout in milliseconds;
  timeout: 40
  #force_sync == true will force mixer to wait for buffers from ALL inputs
  force_sync: true


# lidar data rendering
---
name: ds3d_sensor_fusion_render
type: ds3d::datarender
in_caps: ds3d/datamap
with_queue: sink
custom_lib_path: libnvds_3d_gles_ensemble_render.so
custom_create_function: NvDs3D_CreateGlesEnsembleRender
gst_properties:
  sync: True
  async: False
  drop: False
config_body:
  # window size
  window_width: 1280
  window_height: 480
  color_clear: true
  window_title: DS3D-Lidar-V2X-Fusion-Jetson
  render_graph:
    # cam_0:
    - texture3d_render:
        layout: [0, 0, 640, 480]
        max_vertex_num: 6
        color_clear: false
        texture_frame_key: DS3D::ColorFrame_0+0
    - lidar3d_render:
        layout: [0, 0, 640, 480]
        color_clear: false
        lidar_color: [0, 0, 255]
        # original lidar data key
        lidar_data_key: DS3D::LidarXYZI_0
        # lidar_bbox_key: DS3D::Lidar3DBboxRawData_0
        enable_label: True
        element_size: 4
        # project lidar data into image require image size settings
        project_lidar_to_image: true
        image_width: 640
        image_height: 480
        intrinsics_mat_key: DS3D::Cam0_IntrinsicMatrix
        extrinsics_mat_key: DS3D::LidarToCam0_ExtrinsicMatrix
        #z_range: [-100, 100]
        x_range: [-100, 100]
        line_width: 2.0
        font_size: 20
        #y_range: [-100, 100]
    # lidar top view
    - lidar3d_render:
        # layout [x0, y0, x1, y1]
        layout: [640, 0, 1280, 480]
        view_position: [0, 0, -1]
        view_target: [0, 0, 1]
        view_up: [0, -1, 0]
        perspective_near: 0.1
        perspective_far: 100
        # angle degree
        perspective_fov: 80
        # 0 stands for (layout.x1 - layout.x0) / (layout.y1 - layout.y0))
        perspective_ratio: 0.0
        lidar_color: [0, 255, 0]
        # lidar transformed to camera coordinates data key
        #lidar_data_key: DS3D::LidarAlignedXYZIKey
        # original lidar data key
        lidar_data_key: DS3D::LidarXYZI_0+1
        element_size: 4
        color_clear: false

yuweiw · November 4, 2024, 7:41am

There are nothing wrong from your config file and the timestamp from the log.
Which module assigns the timestamps for each frame of your lidar and camera? It is possible that there is a problem when you set the timestamps to the frame from the lidar or the camera.

j_mueller · November 4, 2024, 9:06am

The timestamp for the LiDAR data is set in my custom-written plugin. In the function readDataImpl(), a 40ms timestamp is added to each new measurement.

I do not set the timestamp for the camera data; it is automatically set, presumably by GStreamer or Deepstream. There is an option to set the timestamp for the camera stream using Aravis (argument do-timestamp), but when I do so, the camera stream’s timestamp is ahead, and the stream fails (see timestamp log below).

Lidar TimeStamp: 0
Lidar TimeStamp: 40
Lidar TimeStamp: 80
Play...
Camera TimeStamp: 80
Mixer TimeStamp: 0
Mixer TimeStamp: 40
Mixer TimeStamp: 80
Renderer TimeStamp: 0
Camera TimeStamp: 1075
Renderer TimeStamp: 40
Renderer TimeStamp: 80
Camera TimeStamp: 1076
Camera TimeStamp: 1077
Lidar TimeStamp: 120
Mixer TimeStamp: 1075
Renderer TimeStamp: 1075
Lidar TimeStamp: 160
Lidar TimeStamp: 200
Camera TimeStamp: 1083
WARNING: mixer is running slower than expected (slowest/fastest stream(:0/1) duration(=994.974359ms/40.000000ms)); timeout setting (40.000000ms) may be incorrect

yuweiw · November 4, 2024, 10:49am

We don’t have experience of using ‘aravissrc’. Could you make sure that this camera won’t drop frames?

j_mueller · November 4, 2024, 2:49pm

Do you have an idea how to check this?

yuweiw · November 5, 2024, 2:20am

We suggest you ask the vendor of your camera directly. They need to ensure that no frames are dropped and that the timestamps are continuous in the plugin.

j_mueller · November 5, 2024, 9:04am

Is the source code for libnvds_3d_video_databridge.so available so that I can understand what happens internally?

yuweiw · November 5, 2024, 10:51am

This part is not open source currently. You can refer to ds-plugin-3d-bridge to get more information.

j_mueller · November 6, 2024, 8:51am

Will the function be open source in the future and is there a way in Deepstream to synchronize two sensors with different timestamps? For example, if Sensor 1 started recording data 2 seconds before Sensor 2 and running 2 seconds ahead with the timestamp?

yuweiw · November 7, 2024, 1:43am

There are no plans to open source for this part at the moment.
We use nvds3dmixer to synchronize two sensors with different timestamps.

yingliu · December 5, 2024, 2:38am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

system · December 19, 2024, 2:39am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
I was using the DeepStream-3D Multi-Modal V2XFusion case and realized that it only supports offline data fusion? DeepStream SDK deepstream	33	57	January 14, 2025
dwImage_getTimestamp return delay by 2 minutes timestamp DRIVE AGX Orin General driveworks	25	190	January 3, 2025
Delay, randomness and dropped frames in RTSP output Stream DeepStream SDK	22	3835	October 12, 2021
RTSP latency does not work with NVSTREAMMUX DeepStream SDK nvbugs	37	5215	January 23, 2022
Running DeepStream 3d depth camera using D455 realsense camera DeepStream SDK	6	264	July 23, 2024
Jetson nano rtsp stream failed with intel realsense 415 usb camera DeepStream SDK	35	5257	October 12, 2021
Deepstream-app custom CSI v4l2src Camera frame drop DeepStream SDK	6	1264	October 12, 2021
Can I create depth image from NvBufSurface? DeepStream SDK	60	42	January 7, 2025
Deepstream performance with 6 rtsp cameras DeepStream SDK hw , gstreamer	16	566	October 12, 2021
Low camera frame rate DeepStream SDK camera , nvbugs	30	6227	October 12, 2021

Delay in Visualization of Camera and LiDAR Data

Related topics