Using multiple cameras with NvRosSIPL and test_egl_cuda_io_multistream

Autonomous Vehicles DRIVE AGX Xavier DRIVE AGX Xavier General
1

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other

Hi all,
As requested, I’ve split my first topic into separate topics. Here’s the excerpt regarding the issues we faced when trying to stream multiple cameras using NvRosSIPL and test_egl_cudi_io-

Type error when instantiating multi-camera streams using NvRosSIPL and invoking test_egl_cuda_io_multistream for some processing (MultiStreamDisplay in this case)-

Not sure if this is expected to work, but we effectively did this by instantiating test_nvros_sipl node with camMask = "0x1111 0x0000 0x0000 0x0000", on another node invoked test_egl_cuda_io_multistream with modified socket paths to match those created by NvRosSIPL, and finally an nvm_eglstream_out node for the display. On execution, there were a couple of errors originating from NvMedia2DBlitEx (I think. Image attached). Where I think this broke down is in the fact that NvRosSIPL posts images of type NvMediaImage but test_egl_cuda_io_multistream expects images of type CUarray. Is there a workaround for this- what seems to be a type conversion problem?

image
image1248×702 96.4 KB

Thanks,
Ken

3

Hi again,

I’ve just found that the issue here is with test_nvros_sipl failing when provided with a camera mask that has more than one camera enabled. We tried this with the multi encoder node. Perhaps you may be able to reproduce this error and check to see if you come across the same errors?

Post build and copy to target, the commands we are running are-

$ NVROS_TOP=/opt/nvidia/nvros/install_isolated
$ cd $NVROS_TOP
$ source setup.bash
$ roscore &
$ nvros_master -c $NVROS_TOP/etc/nvros_master/nvros-multi-enc.nvroscfg  ## Note: nvros-multi-enc.nvroscfg is a custom config (as detailed below)

Where nvros-multi-enc.nvroscfg is-

<node-description-start>
exe = test_nvros_sipl
## Note: test_nvros_sipl.cpp was modified to accept camConfig and camMask as an argument.
## This change may also be hard-coded with the suitable config and mask strings
arg = -c
arg = SF3324_DPHY_x4
arg = -m
arg = 0x1111 0x0000 0x0000 0x0000
<node-description-end>
<node-description-start>
exe = multi_videnc
arg = -n
arg = 4
arg = -a
arg = /tmp/nvros_cam_0
arg = -b
arg = /tmp/nvros_cam_1
arg = -c
arg = /tmp/nvros_cam_2
arg = -d
arg = /tmp/nvros_cam_3
<node-description-end>

The terminal output (with errors) is-
Note: Because of the way we have set things up, you may find that the config path is slightly different. We have also added some print statements into the code for the purpose of debugging.

$ nvros_master -c $NVROS_TOP/../nvros_master_configs/nvros-multi-enc.nvroscfg
[ INFO] [1698160989.392674187]: Config file path : /opt/nvidia/nvros/install_isolated_hostbuild_3_7/../nvros_master_configs/nvros-multi-enc.nvroscfg 

[ INFO] [1698160989.395154693]: Number of nodes to be launched : 2

[ INFO] [1698160989.397718555]: Child process, pid:- 13200 & name:- test_nvros_sipl 

[ INFO] [1698160989.399060804]: Child process, pid:- 13201 & name:- multi_videnc 

[ INFO] [1698160989.444714348]: config: SF3324_DPHY_x4

0x11110x00000x00000x0000[ INFO] [1698160989.445058269]: 
camMask: 0x1111 0x0000 0x0000 0x0000

test_nvros_sipl.cpp: main0 
[ INFO] [1698160989.481152591]: main: No of eglstreams : 4

test_nvros_sipl.cpp: main5, RGBA =  0
[ INFO] [1698160989.484229264]: main: EGLStream socket path : /tmp/nvros_cam_0

[ INFO] [1698160989.484316237]: main: EGLStream socket path : /tmp/nvros_cam_1

[ INFO] [1698160989.484383306]: main: EGLStream socket path : /tmp/nvros_cam_2

[ INFO] [1698160989.484470694]: main: EGLStream socket path : /tmp/nvros_cam_3

[ INFO] [1698160989.505770585]: main: Initing encoder

[ INFO] [1698160989.510772747]: EGLStreamReceivefd: listen_fd: 34

[ INFO] [1698160989.532256087]: WindowSystemWindowInit: Use first available display

WindowSystemWindowInit: err = 0 (NVROS_SUCCESS)/n[ INFO] [1698160989.534119210]: main: Starting encoder threads

[ INFO] [1698160989.534842604]: main: Waiting for encoder threads to complete

[ INFO] [1698160989.535636107]: EGLStreamSendfd: sock_fd: 36

[ INFO] [1698160989.535886849]: EGLStreamSendfd: sock_fd: 36

[ INFO] [1698160989.536050426]: Waiting for EGL stream producer
 
[ INFO] [1698160989.536164246]: EGL Stream consumer - Latency: 16000 usec

[ INFO] [1698160989.536267409]: EGL Stream consumer - Timeout: 16000 usec

[ INFO] [1698160989.536331951]: EncodeFrames: EglStream[0] initialization done

[ INFO] [1698160989.536663969]: EGLStreamSendfd: sock_fd: 37

[ INFO] [1698160989.536877368]: Waiting for EGL stream producer
 
[ INFO] [1698160989.537126254]: EGLStreamSendfd: sock_fd: 43

[ INFO] [1698160989.537310342]: Waiting for EGL stream producer
 
NvRosSIPL.cpp: NvRosErr CNvRosSIPL::NvRosSIPLInitEglStream check_7.3.2: colorSpace = RGBA -  0
[ INFO] [1698160989.540024311]: EGLStreamReceivefd: listen_fd: 34

[ INFO] [1698160990.536478052]: EGL Stream consumer - Latency: 16000 usec

[ INFO] [1698160990.536613537]: EGL Stream consumer - Timeout: 16000 usec

[ INFO] [1698160990.536707134]: EncodeFrames: EglStream[1] initialization done

NvRosSIPL.cpp: NvRosErr CNvRosSIPL::NvRosSIPLInitEglStream check_7.3.2: colorSpace = RGBA -  0
[ INFO] [1698160990.538734916]: EGLStreamReceivefd: listen_fd: 18

[ INFO] [1698160991.537784025]: EGL Stream consumer - Latency: 16000 usec

[ INFO] [1698160991.537901015]: EGL Stream consumer - Timeout: 16000 usec

[ INFO] [1698160991.537966646]: EncodeFrames: EglStream[2] initialization done

NvRosSIPL.cpp: NvRosErr CNvRosSIPL::NvRosSIPLInitEglStream check_7.3.2: colorSpace = RGBA -  0
[ INFO] [1698160991.540424623]: EGLStreamReceivefd: listen_fd: 18

[ INFO] [1698160992.537642385]: EGL Stream consumer - Latency: 16000 usec

[ INFO] [1698160992.537774576]: EGL Stream consumer - Timeout: 16000 usec

[ INFO] [1698160992.537853263]: EncodeFrames: EglStream[3] initialization done

NvRosSIPL.cpp: NvRosErr CNvRosSIPL::NvRosSIPLInitEglStream check_7.3.2: colorSpace = RGBA -  0
MAX96712: Revision 2 detected
MAX96712: Enable periodic AEQ on Link 0
MAX96705: Pre-emphasis set to 0xaa
MAX96705: Revision 1 detected!
Sensor AR0231 RCCB Rev7 detected!
MAX96712: Enable periodic AEQ on Link 1
MAX96705: Pre-emphasis set to 0xaa
MAX96705: Revision 1 detected!
Sensor AR0231 RCCB Rev7 detected!
MAX96712: Enable periodic AEQ on Link 2
MAX96705: Pre-emphasis set to 0xaa
MAX96705: Revision 1 detected!
Sensor AR0231 RCCB Rev7 detected!
MAX96712: Enable periodic AEQ on Link 3
MAX96705: Pre-emphasis set to 0xaa
MAX96705: Revision 1 detected!
Sensor AR0231 RCCB Rev7 detected!
Module_id 22 Severity 6 : NvMediaICPCreateEx 76
Module_id 22 Severity 6 : T19x VI version  0x000019
NvRmStream: Channel submission failed (err=196623)
NvRmStream: Flush failed (err=196623)
Module_id 47 Severity 2 : Finishing task submit: Flush error 196623
Module_id 24 Severity 2 :  Sync point is invalid
Module_id 48 Severity 2 : Failed to set EOF fences
[ERROR] [1698160995.663374830]: test_nvros_sipl NvRosSIPLPostImgToEglStream Error in 2D-Blit: 1 843

[ERROR] [1698160995.663516622]: test_nvros_sipl OnFrameAvailable Error in posting image to eglstream for senor:3 77

[ERROR] [1698160995.663859150]: test_nvros_sipl ImageQueueThreadFunc OnFrameAvailable failed. (status:127)
 126

NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 1 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 1 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 2 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 2 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 1, 3, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 1, 0, 1 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 3 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 3 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 1, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 4 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 4 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 5 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 5 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 1, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 6 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 6 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 7 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 7 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 1, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 8 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 8 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 3, 9 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 3, 9 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 2, 0, 0 
NvRmStream: Flush failed (err=11)
Module_id 47 Severity 2 : Finishing task submit: Flush error 11
Module_id 24 Severity 2 :  Sync point is invalid
Module_id 48 Severity 2 : Failed to set EOF fences
[ERROR] [1698160995.817806021]: test_nvros_sipl NvRosSIPLPostImgToEglStream Error in 2D-Blit: 1 843

[ERROR] [1698160995.817929701]: test_nvros_sipl OnFrameAvailable Error in posting image to eglstream for senor:1 77

[ERROR] [1698160995.818016646]: test_nvros_sipl ImageQueueThreadFunc OnFrameAvailable failed. (status:127)
 126

NvRmStream: Flush failed (err=11)
Module_id 47 Severity 2 : Finishing task submit: Flush error 11
Module_id 24 Severity 2 :  Sync point is invalid
Module_id 48 Severity 2 : Failed to set EOF fences
[ERROR] [1698160995.818621254]: test_nvros_sipl NvRosSIPLPostImgToEglStream Error in 2D-Blit: 1 843

[ERROR] [1698160995.818778662]: test_nvros_sipl OnFrameAvailable Error in posting image to eglstream for senor:2 77

[ERROR] [1698160995.818938662]: test_nvros_sipl ImageQueueThreadFunc OnFrameAvailable failed. (status:127)
 126

NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
^CReceived signal: 2. Quitting
NvRosSIPL.cpp: NvRosSIPLPostImgToEglStream check_10 (CamId, Status, Retry): 0, 0, 0 
$ free(): double free detected in tcache 2
^C
$

You may notice from our debug statement that after the errors only one camera is streaming. Furthermore, we experimented a little and for that the errors above are not found when running the same config but using only ONE camera irrespective of which camera is chosen. However, the error occurs when two, three or four are used.

Look forward to hearing from you,

Thanks,
Ken

4

Based on the following code snippet of test_nvros_sipl, it appears the code is designed to work with a single camera. The camera mask is set to enable only one camera (0x0001) while disabling all other cameras (0x0000 0x0000 0x0000). This explains why you are encountering errors when trying to use multiple cameras.

    /*Define camera config and camera mask as std::string*/
    std::string config("SF3325_DPHY_x4");
    std::string camMask("0x0001 0x0000 0x0000 0x0000");

    /*Init eglstream for camera config and camera masks*/
    err = mNvRosSIPL->NvRosSIPLInitEglStream(camMask, RGBA);
    if(err != NVROS_SUCCESS)
    {
        NVROS_DIAGNOSTICS(ERROR,"Init EglStream failed\n");
        err = NVROS_ERR;
        goto exit;
    }

    /*Register a callback function to receive SIPL errors*/
    mNvRosSIPL->NvRosSIPLRegisterErrCallback(errCallbackFunc);

    /*Initiate the capture process*/
    err = mNvRosSIPL->NvRosSIPLStartCamCap(config, camMask, RGBA);
    if(err != NVROS_SUCCESS)
    {
        NVROS_DIAGNOSTICS(ERROR,"Start camera capture failed\n");
        err = NVROS_ERR;
        goto exit;
    }
5

I would hope that’s not the case. I say that because to me it seems like the script test_nvros_sipl is simply invoking functions from NvRosSIPL. I’m assuming here that NvRosSIPL has the capabilities to stream multiple cameras? I’ve briefly reviewed the source code and it seems that way.

This brings me back to the questions in my first topic- How else would you suggest we go about streaming multiple cameras simultaneously using NvROS? (assuming nvros_cam_cap_multistream is obsolete?)

Thanks,
Ken

6

test_nvros_sipl is a test application using the ‘nvros_sipl’ library (with its source code available) to set up a single camera. You have to modify it for your use case.

Additionally, please note that DRIVE OS 5.2.6 is the final release for the Xavier generation, and we do not maintain NvROS after that. Therefore, it should be considered a reference only.