- Hardware Platform:4080
- CUDA:12.6
- Deepstream:7.1
I was using the DeepStream-3D Multi-Modal V2XFusion case and realized that it only supports offline data fusion!
This case uses libnvds_lidarfileread.so, and the provided open source case only loads offline lidar files, is there a real-time case? I want to access the hardware to use.
One of the parts about 2D-3D camera vision seems to support offline file loading as well, and I want to load rtsp camera data streams.
How should I access my own hardware, when using the DeepStream-3D Multi-Modal V2XFusion case?
Is there a demo that I can use?
No. Different hardware will have different logic to get data, it should be done with the lidar vendor who developed the driver software for their device. As you have seen that the lidar hardware data reading can be encapsulated with ds3d::dataloader interfaces to make it as the source of the pipeline.
The lidar(radar) device driver should be provided by the vendor(I think the drivers are not open source). You may encapsulate the data reading process with ds3d::dataloader interfaces.
I think you should provide basic libraries for reading lidar online, just like cameras provide demos for serial and rtsp data streaming, most of the vendors of lidar also provide serial communication, like dev/*0 dev/*1
libnvds_lidarfileread.so has a function to createLidarFileLoader, which is provided in the case to read the offline data, do you mean that I have to modify this function by myself to achieve reading the real-time lidar data?
Yes.
Something is different. There is camera device open source interface such as V4L2 Part I - Video for Linux API — The Linux Kernel documentation in linux, the apps can reach any camera’s data because most camera vendors provide their driver as V4L2 interfaces but not just a node as /dev/video*.
The RTSP is standard RETF RFC protocol RFC 2326 - Real Time Streaming Protocol (RTSP), any RTSP server should obey this standard protocol so that any RTSP client(stream player) can access and parsing the stream without knowing who set the server or how is the server set.
Do you mean there is a standard Lidar device software API protocol which every Lidar vensor will develop their driver to expose the same APIs? If so, can you introduce the protocol here and we can study whether we can develop the Lidar device reader based on this.
One can refer to lidarscan in ros, as most lidar are readable using rivz in ros, and from that point on most lidars follow a standard data format.
How long does it take if you do it? If the cycle time is long, give me some hints and I’ll try to do it.
So you are asking a sample for lidar reader with ROS? Why do you want to use V2XFusion model in ROS? Can you introduce your requirements? Which company and which project are you working for?
ros can support lidar, so the protocol of lidar should be a standard protocol, you can refer to ros to receive the lidar data protocol stream.
No.
I prefer to use deepstream’s pipeline library to get the lidar streams because it works well and is easy to post-process.
How long is your development cycle in general, if it’s too long, I’ll try to develop it, you can give me some demos and hints.
I’ve been using deepstream for a long time, from 6.0 to now 7.1, and now with the addition of tritonServer and lidar , and really like the functionality and the way it is used. Doing some cases of recognition myself, in order to be able to get a better job.
They all use the common ROS node interfaces. But the nodes are developed by the lidar vendors themselves. E.G. Slamtec/rplidar_ros is the Slamtec lidar ROS node implementation which is developed and maintained by Slamtec but not ROS.
It is not a good idea to port the ROS to Linux(ubuntu). And ROS is a OS level framework, DeepStream can’t run on it directly.
DeepStream has defined the common DS3D interfaces, the users can implement their own data source which comply with the DS3D interfaces.
The local file reader is the sample of how to use the ds3d::dataloader interfaces. You have already got the source code for reference.
So what format should I convert the vendor’s customized lidar data into that will be recognized by deepstream? As in your offline case, it should be parsing the offline lidar data into a format that is post-processed by deepstream’s pipeline, is there a documentation for that?
The format is decided by the model. Lidar_AI_Solution/CUDA-V2XFusion at master · NVIDIA-AI-IOT/Lidar_AI_Solution
The /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/v2xfusion/scripts/v2x_data_prepare.py shows how we prepare the DAIR-V2X dataset to the format needed by DeepStream preprocessing.
Now is there a lidar data is sent out via kafka communication protocol, can I access the data to datafilter without using dataloader, using kafka from deepstream? Are there any libraries that can be used in between? What I know so far is that there are already kafka and datafilter, is it possible to design a solution like this? What should I change?
kafka->xxx-“what lib”->datafilter-----
No, there is no such sample now even we support Kafka subscribe. The lidar data size is very large, are you sure you want to transfer it through Kafka?
You can implement the Kafka subscribe inside the ds3d::dataloader.
You can implement with the SyncImplDataLoader interface. Please refer to /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/dataloader/lidarsource/
I notice that this py generates three tensors(camera2lidar.tensor\camera_intrinsic.tensor\lidar2camera.tensor), but I don’t see where these three tensors are used, these three tensors are used more importantly, it’s about the program transforming the camera internal reference, the camera to LIDAR transformation matrix
Also I see the internal and external parameters in the configuration file ds3d_lidar_video_sensor_v2x_fusion_single_batch.yml:
I want and know whether to use those three tensors or this inside/outside parameter in the config file? If the three tensors are used, are they used in that program?
name: lidar_alignment_filter
type: ds3d::datafilter
link_to: v2xfusion_inference
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_alignment_datafilter.so
custom_create_function: createLidarAlignmentFilter
config_body:
cam_width: 1920
cam_height: 1080
row_major: True
disable_lidar_data_align: True
filter_input_datamap_key: DS3D::LidarXYZI_0
output_cam_intrinsic_param_keys: [DS3D::Cam0_IntrinsicParm]
output_cam_intrinsic_mat_keys: [DS3D::Cam0_IntrinsicMatrix]
output_lidar_to_cam_extrinsic_mat_keys: [DS3D::LidarToCam0_ExtrinsicMatrix]
cam_intrinsic: [
[2175.554, 0.000, 969.823, 0.000,
0.000, 2320.126, 572.754, 0.000,
0.000, 0.000, 1.000, 0.000]]
lidar_to_cam_extrisic: [
[-0.002, -1.001, -0.000, -3.616,
-0.219, -0.001, -0.806, 5.457,
0.976, -0.007, -0.183, 1.201]]
cam_input_keys: [DS3D::ColorFrame+0]
max_points: 70000
mem_pool_size: 32
align_to_intrinsic: false
lidar_element_size: 4
Only these two files: “intervals.tensor” and “geometrys.tensor” are used in the sample app. Please refer to /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/ds3d_lidar_video_sensor_v2x_fusion.yml
...
name: v2xfusion_inference
type: ds3d::datafilter
link_to: ds3d_sensor_fusion_render
in_caps: ds3d/datamap
out_caps: ds3d/datamap
...
custom_preprocess_lib_path: /opt/nvidia/deepstream/deepstream/lib/libnvds_3d_v2x_infer_custom_preprocess.so
custom_preprocess_func_name: CreateInferServerCustomPreprocess
preprocess:
intervalsFrom: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/v2xfusion/example-data/intervals.tensor
geometrysFrom: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/v2xfusion/example-data/geometrys.tensor
imagesFrom: DS3D::VideoPreprocessTensor+0
featsFrom: DS3D::LidarFeatureTensor+1
coordsFrom: DS3D::LidarCoordTensor+1
NFrom: DS3D::LidarPointNumTensor+1
...
Please refer to /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/inference_custom_lib/ds3d_v2x_infer_custom_preprocess/nvinferserver_custom_preprocess.cpp for how the data are used to generate the tensor inputs of the model.
Why is this happening when I use my own dataset, the point cloud data from the lidar is clipped and just about staggered with the data from the image, it seems that the vehicle enters the image view just about invisible in the lidar point cloud. Why is this? How do I adjust the parameters?
I tried to adjust the following parameters, including: inside and outside of the camera and lidar, as well as the x\y\z range parameter and also the viewpoint related parameters, but none of them work, WHY?
cam_intrinsic: [
[1077, 0.00, 0.00,
0.00, 1071.3, 0.00,
955.3185, 549.4412, 1.00,
0.00, 0.00, 0.00]]
lidar_to_cam_extrisic: [
[0.8222334, 0.2422148, -0.5150381,
0.5691114, -0.3604956, 0.7390231,
-0.00666665, -0.9007635, -0.4342587,
-0.068337, 0.3206510, 2.884617]]
z_range: [-300, 300]
x_range: [-1000, 1600]
y_range: [-1000, 1600]
view_position: [20, 70, 10]
view_target: [30, 10, 10]
Can you post a video for it for us to understand the thing you described?
at 6s-10s