Recording issue about CAN messages through Aurix

Hello, I am trying to record CAN message from Aurix and the camera video simultaneously.
I connected my can cable to can1 port and accessed from Aurix can-a,
and can message from only 1 can ID should be transfered by Aurix.

I used the ‘recorder’ file to record can message,
but it met error and stoped frequently after the recording is started.

Where is the problem?

The messages are as following.

nvidia@tegra-ubuntu:~/odaiba/record_test$ ./recorder ./rig_config/rig_aurix_can_a_1c.json
[29-10-2019 10:58:54] Platform: Detected DDPX - Tegra B
[29-10-2019 10:58:54] Initialize DriveWorks SDK v2.0.2081
[29-10-2019 10:58:54] Release build with GNU 4.9.4 from heads/buildbrain-branch-0-g36b127f against Drive PDK v5.1.0.2
[29-10-2019 10:58:54] TimeSource: monotonic epoch time offset is 1572341288367830
[29-10-2019 10:58:54] PTP Time is available from NVPPS Driver
[29-10-2019 10:58:54] Platform: number of GPU devices detected 1
[29-10-2019 10:58:54] Platform: currently selected GPU device integrated ID 0
[29-10-2019 10:58:54] SDK: Resources mounted from /usr/local/driveworks-2.0/data/
[29-10-2019 10:58:54] SDK: Create NvMediaDevice
[29-10-2019 10:58:54] egl::Display: found 2 EGL devices
[29-10-2019 10:58:54] egl::Display: use drm device: drm-nvdc
[29-10-2019 10:58:54] Platform: currently selected GPU device integrated ID 0
[29-10-2019 10:58:54] Loading rig file: ./rig_config/rig_aurix_can_a_1c.json
[29-10-2019 10:58:54] SensorFactory::createSensor() -> can.aurix, ip=10.42.0.146,bus=a,config-file=EasyCanConfigFileA.conf,file-buffer-size=8192
[29-10-2019 10:58:54] SensorFactory::createSensor() -> camera.gmsl, camera-type=ar0231-rccb-bae-sf3324,camera-group=a,camera-count=1,siblingIndex=0,slave=0,format=lraw,output-format=raw,fifo-size=30,file-buffer-size=4194304
[29-10-2019 10:58:54] CameraGMSL: Create NvMediaIPPManager
[29-10-2019 10:58:54] CameraGMSL: required FPS = 30, resolution = 1920x1208
nvmedia isc: IscPwrCtrlInit: 939: Camera power control library: NVCCP
Max96712 Rev 2 detected!
MAX96712: Enable periodic AEQ on Link 0
MAX96705: Pre-emphasis set to 0xaa
Sensor AR0231 RCCB Rev7 detected!
[29-10-2019 10:58:55] CameraGMSL: default IPP event callback function set
[29-10-2019 10:58:55] camera params: index_table=off,file=,camera-type=ar0231-rccb-bae-sf3324,camera-group=a,camera-count=1,siblingIndex=0,slave=0,format=lraw,output-format=raw,fifo-size=30,file-buffer-size=4194304
[29-10-2019 10:58:55] CameraGMSL: serializer bitrate not specified. Using 8000000.
[29-10-2019 10:58:55] EGLStream: EGL_SUCCESS
[29-10-2019 10:58:55] StreamProducerCUDA: successfully initialized
Rig: rig_aurix_can_a_1c.json NewSink: /dev/null
[29-10-2019 10:58:55] EndpointUDP: started 10.42.0.146:50000
[29-10-2019 10:58:56] SensorManager: started
Press s<Enter> to start, just <Enter> to see progress, q<Enter> to quit.
[29-10-2019 10:58:56] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388704744, id=259, length=38
[29-10-2019 10:58:56] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:58:56] CAN messages [291] for 0 at 1572346
[29-10-2019 10:58:56] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
[29-10-2019 10:58:56] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388706354, id=259, length=38
[29-10-2019 10:58:56] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:58:56] CAN messages [291] for 0 at 645699600000000
[29-10-2019 10:58:56] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
[29-10-2019 10:58:58] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388707964, id=259, length=38
[29-10-2019 10:58:58] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:58:58] CAN messages [291] for 0 at 4026536468026536
[29-10-2019 10:58:58] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
[29-10-2019 10:58:59] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388709564, id=259, length=38
[29-10-2019 10:58:59] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:58:59] CAN messages [291] for 0 at 4026536468026536
[29-10-2019 10:58:59] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
[29-10-2019 10:59:01] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388711174, id=259, length=38
[29-10-2019 10:59:01] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:59:01] CAN messages [291] for 0 at 1572346
[29-10-2019 10:59:01] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
s
[29-10-2019 10:59:03] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388712784, id=259, length=38
[29-10-2019 10:59:03] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:59:03] CAN messages [291] for 0 at 327680004449484
[29-10-2019 10:59:03] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
[29-10-2019 10:59:03] camera params: file=/home/nvidia/odaiba/record_test/camera:front:center:120fov.lraw,camera-type=ar0231-rccb-bae-sf3324,camera-group=a,camera-count=1,siblingIndex=0,slave=0,format=lraw,output-format=raw,fifo-size=30,file-buffer-size=4194304
[29-10-2019 10:59:03] CameraGMSL: serializer bitrate not specified. Using 8000000.
[29-10-2019 10:59:04] EGLStream: EGL_SUCCESS
[29-10-2019 10:59:04] StreamProducerCUDA: successfully initialized
[29-10-2019 10:59:04] EncoderLRAW: End of recording, Camera Frames = 239 Encoded Frames =  239 Dropped Frames =  0
[29-10-2019 10:59:04] EGLStream: EGL_SUCCESS
Rig: rig_aurix_can_a_1c.json NewSink: .
Rig: rig_aurix_can_a_1c.json Sensor: can:vehicle Bytes: 54
Rig: rig_aurix_can_a_1c.json Sensor: camera:front:center:120fov Bytes: 10879700
Rig: rig_aurix_can_a_1c.json EndOfSensorsInfo
[29-10-2019 10:59:04] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388714394, id=259, length=38
[29-10-2019 10:59:04] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:59:04] CAN messages [291] for 0 at 327680004449484
[29-10-2019 10:59:04] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
[29-10-2019 10:59:06] Received from 10.42.0.146:50000 - 46 bytes, src=4, dst=2, app=202, time=388715994, id=259, length=38
[29-10-2019 10:59:06] Received at least 1 CAN messages within 22 bytes
[29-10-2019 10:59:06] CAN messages [291] for 0 at 1572346
[29-10-2019 10:59:06] Parsed 1 CAN messages, found 1 messages on selected EasyCAN bus.
[29-10-2019 10:59:06] IndexTable: cannot populate index table, sensor events are non-monotonic. Expecting timestamp for event 2 larger or equal to 327680004449484 found 1572346. Seeking range will be limited to: events[0,0], time[327680004449484,327680004449484].
terminate called after throwing an instance of 'dw::core::Exception'
  what():  DW_END_OF_STREAM: SensorSerializerByte: error updating seek table
Aborted (core dumped)

Dear Liang_Qi,

Could you share the json file you used? Thanks.

Thanks for the replay.
My json file is as below.

{
    "rig": {
        "sensors": [
            {
                "name": "can:vehicle",
                "nominalSensor2Rig_FLU": {
                    "roll-pitch-yaw": [
                        0.0,
                        0.0,
                        0.0
                    ],
                    "t": [
                        1.77,
                        0.0,
                        1.47
                    ]
                },
                "parameter": "ip=10.42.0.146,bus=a,config-file=EasyCanConfigFileA.conf,file-buffer-size=8192",
                "properties": null,
                "protocol": "can.aurix"
            },
            {
                "name": "camera:front:center:120fov",
                "nominalSensor2Rig_FLU": {
                    "roll-pitch-yaw": [
                        0.0,
                        0.0,
                        0.0
                    ],
                    "t": [
                        1.77,
                        0.0,
                        1.47
                    ]
                },
                "parameter": "camera-type=ar0231-rccb-bae-sf3324,camera-group=a,camera-count=1,siblingIndex=0,slave=0,format=lraw,output-format=raw,fifo-size=30,file-buffer-size=4194304",
                "properties": {
                    "Model": "ftheta",
                    "bw-poly": "0 0.0010337684070691 2.2048213921266e-08 5.6866707523473e-12 3.1827639075709e-14",
                    "cx": "964.875977",
                    "cy": "645.572449",
                    "height": "1208",
                    "width": "1920"
                },
                "protocol": "camera.gmsl"
            }
        ],
        "vehicle": {
            "valid": true,
            "value": {
                "COMMENT": "steeringCoefficient is not validated",
                "aeroHeight": 1.5,
                "aerodynamicDragCoeff": 0.7,
                "axlebaseFront": 1.582,
                "axlebaseRear": 1.575,
                "brakeActuatorTimeConstant": 0.5,
                "brakeForceOutput": [
                    0,
                    714,
                    1423,
                    2143,
                    2857,
                    3571,
                    4286,
                    5000,
                    5714,
                    6429,
                    7143,
                    7857,
                    8571,
                    9286,
                    10000
                ],
                "brakeStateInput": [
                    0,
                    0.07,
                    0.14,
                    0.21,
                    0.29,
                    0.36,
                    0.43,
                    0.5,
                    0.57,
                    0.64,
                    0.71,
                    0.79,
                    0.86,
                    0.93,
                    1.0
                ],
                "bumperFront": 0.912,
                "bumperRear": 1.109,
                "centerOfMassHeight": 1.5,
                "centerOfMassToFrontAxle": 1.286,
                "centerOfMassToRearAxle": 1.564,
                "driveByWireTimeConstant": 0.26,
                "driveByWireTimeDelay": 0.11,
                "frontCorneringStiffness": 30654.0,
                "frontalArea": 1.0,
                "height": 1.473,
                "inertia": 1780.8,
                "length": 4.872,
                "mass": 1779.4,
                "maxEnginePower": 134000.0,
                "name": "ford fusion 2017",
                "rearCorneringStiffness": 36407.0,
                "rollingResistanceCoeff": 0.015,
                "steeringCoefficient": 14.8,
                "throttleActuatorTimeConstant": 1.0,
                "throttleForceOutput": [
                    0,
                    312,
                    623,
                    1086,
                    2646,
                    4272,
                    5370,
                    6052,
                    6363,
                    6556,
                    6730,
                    6905,
                    7092,
                    7283,
                    7473
                ],
                "throttleStateInput": [
                    0,
                    0.07,
                    0.14,
                    0.21,
                    0.29,
                    0.36,
                    0.43,
                    0.5,
                    0.57,
                    0.64,
                    0.71,
                    0.79,
                    0.86,
                    0.93,
                    1.0
                ],
                "wheelDiameter": 0.673,
                "wheelbase": 2.85,
                "width": 1.852,
                "widthWithMirrors": 2.121
            }
        }
    },
    "version": 2
}

The EasyCanConfigFileA.conf file is as below.

MsgId   : 0x123
CtrlId  : EC_CAN_BUS_A
FrmTyp  : CAN_ID_STANDARD
DestId  : TEGRA_DEVICE_A
DestSwc : SWC_ID_CANDATA

MsgId   : 0x123
CtrlId  : EC_CAN_BUS_A
FrmTyp  : CAN_ID_STANDARD
DestId  : TEGRA_DEVICE_B
DestSwc : SWC_ID_CANDATA

Anyone could give me some comment or advice ?
Thanks.

Dear Liang_Qi

Sorry for late update.
Can you please file a bug for this topic?
Please share bug ID, we will look into this issue.
Please login to https://developer.nvidia.com/drive with your credentials. Please check MyAccount->MyBugs->Submit a new bug to file bug. Thanks.

Thanks for your reply.
I have reported this bug and the bug ID is 2756661.

As it is a bug, does it mean that I can not get any solution until next release?
Is there any other way to continue my development?

Hi Liang_Qi,

You can try to connect to CAN2/6 and then access it with can0/1 setting as below.

{
                "name": "can:vehicle",
                "nominalSensor2Rig_FLU": {
                    "roll-pitch-yaw": [
                        0.0,
                        -0.0,
                        0.0
                    ],
                    "t": [
                        0.0,
                        0.0,
                        0.0
                    ]
                },
                "parameter": "device=can0",                                                                                                                                                                                           
                "properties": null,
                "protocol": "can.socket"
            },

You can check below sections in “DriveWorks SDK Reference” for details.

  • Rig File Format
  • Sensor Enumeration Sample

Thanks for the advice. Such issue don’t ocurr when directly using can 2/6.
But I have 3~4 can ports to record while Xavier can directly access only 2 of them.
As a result, access can through Aurix is inevitable in my development.

Do you see any issue if record only AurixCAN data (no camera)?
If yes, how about using other samples? e.g. sample_canbus_logger, sample_record.
Thanks!

I test it again and find the following.

Record only AurixCAN data:
Although some timestamp shown on the screen may be strange, recording using ‘recorder’ continues until user stops.
When using ‘replayer’ to replay, the replay speed is much faster than realtime sequence.

When using ‘sample_record’ to record only can1, non-monotonic error also occurs, but the recording continues. The replay speed is much faster than realtime sequence.

Record both AurixCAN and Camera:
The recording using ‘recorder’ is stopped by the non-monotonic error. This issue occurs with probability of 50% for 20s recording.
For successful recordings, when using ‘replayer’ to replay, the can data and camera video freezed at the first data/frame.
The replay of only camera video is OK.
The replay of only can data freezed at the first data.

Thank you for the information! I’ll check with our QA team and then get back to you here.

BTW, Drive Software 10.0 will be officially announced soon but actually it’s already accessible via SDK Manager now. So you can also check the issue with the new release.

Thanks for your advice.
But it is not easy for me to update to Drive Software 10.0,
because I have to investigate the compatiblility of the new version and make sure it won’t affect other member’s development.
It is a difficult problem for me and may cost a lot of time,
it will be helpful if there are some official documents for the compatibility.

By the way, did this timestamp issue occur on your environment?

For the non-monotonic error, we suspect it’s due to an issue in handling hardware timestamps from CAN messages.

Could you help try with sample_canbus_logger to see if able to reproduce?
If reproducible, run the sample with “–hwtime=0” to disable hardware timestamps and then see if helps.

For Drive Software 10.0 Documentation (will be available at https://developer.nvidia.com/drive/documentation soon), “SDK Porting Guide” section will be included in its “DriveWorks SDK Reference”. We hope it will help on your moving to the release.

sample_canbus_logger continues without any error, but I don’t know which parameter is the received timestamp, so I am not sure the timestamp is right or wrong.

I also tried ‘–hwtime=0’, but an ‘Unknow option hwtime’ error is shown for ‘sample_canbus_logger’ and ‘sample_recorder’. Which sample can use this ‘hwtime’ option ?

Hi Liang_Qi,

I disabled hardware timestamp in the attached debugging library. Could you please help to try if still able to reproduce the issue with it? Thanks!
libdriveworks.so.2.0.2088.zip (27.1 MB)

Dear VickNV,

Thanks for your replay.

I want to try this, but I don’t know how to deal with the library file.
Would give me some step by step guide about how to use this debugging library with the ‘recorder’ command ?

On the target, please run below command to find out where is the library and then replace with the attached one (remember to back up the original one). After that, you can try if recorder application still hit the issue.

$ find -L /usr/local/driveworks -iname ‘libdriveworks.so.2.0.2088’

I checked by the command '$ find -L /usr/local/driveworks -iname ‘libdriveworks.so.2.0.2088’ ’
and could not find it.
I checked the folder and found the file ‘libdriveworks.so.2.0.2081’ in my driveworks.
Is it OK to just change the ‘.2088’ you provided to ‘.2081’ in order to fit my driveworks?

I have no ubuntu 16.04 and Drive Software 9.0 installed on my side so I was not aware of the different version.
Yes, please help rename to 2081 and have it a try. Thanks!

I have tried by renaming the ‘.2088’ file to ‘.2081’ and replacing the original file.

The recording of only aurix can signal is no longer interrupted by error.
But when using ‘replayer’ to replay, the replay speed is still much faster than realtime sequence.

Another problem is that the recording of both aurix can signal and camera is still interrupted by the non-monotonic error.

It seems that although the timestamp from aurix is discarded, timestamp from xavier is not used.
Would you give me some further advices? Thanks.