Connecting camera.nvidia-ip

Autonomous Vehicles DRIVE AGX Xavier DRIVE AGX Xavier General
,
1

I’m running a simulator and I’m sending the camera sensor frames over the network.
I want to use SAL to create a camera sensor which will recieve the frames over network and I saw there is a camera type “camera.nvidia-ip”.

I haven’t found any documentation on the packet format and how to tell the camera its parameters. How should I send the frames and set the camera parameters?
Currently the simulator just sends raw RGB pixels over the socket.

To test it I wrote a rig file and initialized the camera with the sensor manager

"parameter": "host=127.0.0.1,port=8000,output-format=raw+data,format=raw",
"properties": {
   "Model": "pinhole",
   "fx": "831.3843876330611",
   "fy": "831.3843876330611",
   "height": "540",
   "width": "960",
   "cx": "480",
   "cy": "270",
   "distortion": "0 0"
},
"protocol": "camera.nvidia-ip"

But it doesn’t work

[20-10-2020 14:17:49] SensorFactory::createSensor() -> camera.nvidia-ip, host=127.0.0.1,port=8000,output-format=raw+data,format=raw
[20-10-2020 14:17:49] CameraIP: Waiting for connection to source stream.
[20-10-2020 14:17:49] SocketClient: connected 127.0.0.1:8000
[20-10-2020 14:17:49] SensorManager: DW_SAL_SENSOR_ERROR: Protocol failure, packet size must be a power of 2.

Hardware Platform: DRIVE AGX Xavier™ Developer Kit
Software Version: DRIVE Software 10
Host Machine Version: native Ubuntu 18.04

3

Hi @biber.barak,

You can use “data.socket” sensor protocol as listed in below sample and mentioned in below topic.

nvidia@tegra-ubuntu:/usr/local/driveworks/bin$ ./sample_sensors_info
...
Platform: OS_DRIVE_V5L - CURRENT:
...
   Sensor [10] : data.socket ? ip=X.X.X.X,port=XXXX
...
1 Like
6

Hi @biber.barak,

the camera.nvidia-ip protocol is an experimental one for internal usage. that is why you haven’t found any documentation on it.

you should build your interface based on the usage of protocol “data.socket” that is open for any data type you want.

There is no “ready to use” sample on how to receive images with DriveWorks from etherent.
But I have created a basic template sample code for you and your team to start with – that sample sends locally synthetic pattern images from the source on the host computer to the destination on the same host (127.0.0.1).

I’ve attached the basic sample code (main.cpp that can be compiled as one of the DW samples and needs to be extended and adjusted to your use-case and image protocol) that reads an image from TCP and displays it and a simple tcp server(tcpServer.cpp that simply compiles as a stand along program) that sends a simple RGB pattern image.
main.cpp (11.8 KB) tcpServer.cpp (2.0 KB)

1 Like
7

Thank you
I do have a working server client using the data.socket sensor I was just experimenting with the nvidia-ip to make it more like the real camera.
I had another problem using data.socket.
Can it work with the dwSensorManager because I initialized the manager with the data.socket sensor, GPS and IMU but I don’t seem to receive events for the data sensor only for the GPS and IMU
Thank you

9

It still doesn’t work

[ 21-10-2020 13:05:20] SensorFactory::createSensor() -> data.socket, ip=127.0.0.1,port=8000
[21-10-2020 13:05:20] DataSensor: Finished initializing DataSensor
Camera image with 960x540
[21-10-2020 13:05:20] SocketClient: connected 127.0.0.1:8000
[21-10-2020 13:05:20] SensorManager: started

In the onProcess method I call

        const dwSensorEvent* event = nullptr;
        auto status                        = dwSensorManager_acquireNextEvent(&event, 10, m_sensor_manager);
        if (status == DW_TIME_OUT)
            return;

        if (status != DW_SUCCESS)
        {
            if (status != DW_END_OF_STREAM)
                printf("Error reading sensor %s\n", dwGetStatusName(status));
            else
            {
                handleEndOfStream();
                break;
            }
        }  
        std::cout << "dwSensorManager_acquireNextEvent event " << event->type << std::endl;

But It never prints. (it did when the rig contained the GPS and IMU)

10

Hello @biber.barak,

sorry for the confusion, sensor manager have no implementation for data type sensor events on the latest release - at current time that is DW 2.2.
In the future release this will be fixed.

At the meantime, please try and use the former solution I suggested, or try and use the sensor manager but for data sensor use the dwSensor_readRawData api.

For your convenience, I’ve prepared a basic sample application similar to what I’ve posted above with the change of using dwSensorManager and the corresponding rig file.
main.cpp (12.5 KB)
rig.json.txt (1.6 KB)

1 Like