Recording Radar Data from custom plugin

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.7.1.8928
other

Host Machine Version
native Ubuntu 18.04
other

Hi,

some time ago we wrote a custom Radar sensor plugin and it’s working fine with the radar replay sample.

But when trying to record data from our radar there are errors.
I debugged with the sample_record application and saw, that

status = dwSensorSerializer_serializeDataAsync(data, size, serializer);

throws the exception when inserting data from our plugin.

Here’s the complete output:

[09-12-2021 09:47:40] Platform: Detected Generic x86 Platform
[09-12-2021 09:47:40] TimeSource: monotonic epoch time offset is 1639042230718887
[09-12-2021 09:47:40] Platform: number of GPU devices detected 1
[09-12-2021 09:47:40] Platform: currently selected GPU device discrete ID 0
[09-12-2021 09:47:40] SDK: Resources mounted from /usr/local/driveworks/data/
[09-12-2021 09:47:40] TimeSource: monotonic epoch time offset is 1639042230718887
[09-12-2021 09:47:40] Initialize DriveWorks SDK v2.2.3136
[09-12-2021 09:47:40] Release build with GNU 7.4.0 from heads/buildbrain-branch-0-gca7b4b26e65
[09-12-2021 09:47:40] SensorFactory::createSensor() -> radar.custom, decoding=false,decoder-path=/home/nvidia/dw_samples/build/src/sensors/plugins/radar_plugin_ars408/libradar_plugin_ars408.so,dbc-path=/home/nvidia/dw_samples/fast_config/dbc-and-can/ARS408/dbc/ars408.dbc,device=can-radar,can-protocol=can.socket
createSensor: Return type configuration not specified, setting default (cluster)
[09-12-2021 09:47:40] SensorFactory::createSensor() -> can.socket, device=can-radar
[09-12-2021 09:47:40] CANSocket: Cannot get current state of hardware time stamping: ioctl(SIOCGHWTSTAMP, can-radar) -> No such file or directory
[09-12-2021 09:47:40] CANSocket: software based timestamps will be used for can-radar
[09-12-2021 09:47:40] 
CAN DBC parsed results:
numMessages: 16
numSignals: 136
numSignalsExtendedValueType: 0
numIgnoredMessages: 0
numInvalidMessages: 0
numInvalidSignals: 0
numInvalidSignalsExtendedValueType: 0

[09-12-2021 09:47:40] CANSocket: ioctl(SIOCSHWTSTAMP, can-radar) failed -> No such file or directory. Cannot enable HW timestamps, try with root user.
[09-12-2021 09:47:40] CANSocket: use SW based timestamps for can-radar
[09-12-2021 09:47:40] CANSocket: started can-radar
[09-12-2021 09:47:43] Driveworks exception thrown: DW_INVALID_ARGUMENT: SensorSerializerByte: error updating seek table

There seems to be a problem with the data for the recorder, but again, it’s working fine in the radar replay sample.

One strange thing though:
During development of the plugin I had to work around a “wrong” size of data returned in pushData:

    dwStatus pushData(const uint8_t* data, const size_t size, size_t* lenPushed)
    {
        //FIXME: Size from the framework is always 12 bytes bigger than it should be.
        //       We just subtract 12 and it is working. Find out how this is supposed 
        //       to be done.
        m_buffer.enqueue(data, size-12);
        *lenPushed = size;
        return DW_SUCCESS;
    }

After this “fix”, the whole plugin started to work properly with the replay sample. I could imagine, that this could be the cause of the recording problem. However, I don’t know how to fix it.

For reference, here’s my readRawData()method:

    dwStatus readRawData(const uint8_t** data, size_t* size, dwTime_t* timestamp, dwTime_t /*timeout_us*/)
    {
        dwCANMessage result;
        dwStatus status = DW_SUCCESS;
        status = dwSensorCAN_readMessage(&result, 1000, m_ars_408Radar);
        if (status != DW_SUCCESS)
        {
            return status;
        };
        
        dwTime_t now{};
        status = dwContext_getCurrentTime(&now, m_ctx);
        if (status != DW_SUCCESS)
        {
            std::cerr << "readRawData: error getting timestamp" << std::endl;
            return DW_SAL_SENSOR_ERROR;
        }

        radarMessage* message = nullptr;

        bool ok = m_slot.get(message);
        if (!ok)
        {
            std::cerr << "readRawData: Read raw data, slot not empty\n";
            return DW_BUFFER_FULL;
        }

        message->timestamp = now;
        message->payload_size = sizeof(dwCANMessage);
        message->payload = result;
        
        *data = reinterpret_cast<uint8_t*>(message);
        *size = sizeof(radarMessage);
        *timestamp = now;

        return DW_SUCCESS;
    }   

The radarMessage is defined as follows:

   /// Encapsulated radar CAN message with header.
    #pragma pack(push, 1) // Makes sure you have consistent structure packings.
    typedef struct
    {
        uint32_t payload_size;
        dwTime_t timestamp;
        dwCANMessage payload;
    } radarMessage;
    #pragma pack(pop)

I tried some things with packing etc, but couldn’t find out where the 12 bytes came from.

Are the two things related? What am I doing wrong?

Thanks in advance.

EDIT: I changed the recording sample to subtract also the 12 bytes… So the twelve byte problem should also solve the recording problem…

Dear @f.petry,
Did subtracting 12 bytes in recording sample helped?

Yes it did help, but then validatePacket(),which checks the size against the radarMessage struct, fails.
So somewhere, there is a diff between the sizes in the callback functions.

status = dwSensor_readRawData(&data, &size, 500000, radarSensor);
from the record sample application returns a size that is 12 higher than the data size is actually. In the readRawData() method of the plugin the radarMessage is 90 bytes and the *size is set accordingly. Somewhere between the plugin and the application, the framework add 12 to the size.

It seems, that the framework does not account for the “header” (payload_size, timestamp). The documentation of readRawData() clearly states, that the data shall be output in a struct of this organization, except for CAN sensors. Even if our plugin uses a CAN bus internally, the returned values follow the documentation and something strange is done in the framework.

I changed our plugin to return only the payload size and recording works, but now somewhere in the replay sample there appears a memory corruption:

[06-01-2022 14:35:55] IndexTable: loading from file "/home/nvidia/dw_samples/fast_config/rig-file/_recordings/2021-12-16_Innovationscampus/radar_front_center.bin.seek"
[06-01-2022 14:35:55] Radar: seek table with 33215 entries covering 1062856 events ranging from 1569031087990486 to 1569031428560088 usecs
[06-01-2022 14:35:55] Initialize DriveWorks VisualizationSDK v2.2.3136
[06-01-2022 14:35:55] Initialize DriveWorksGL SDK v2.2.3136
[06-01-2022 14:35:55] GL-SDK: initialize OpenGL
double free or corruption (out)

I think, that somewhere in the framework there is an inconsistency in what the size in the “header” is.

Any other hints/ideas?

From here: HERE I know that you internally have implemented something similar to a plugin for a CAN based ARS430. Maybe you can compare your code and help me out?

Dear @f.petry,
I will check internally and update you on this topic. Is it possible for you test with latest DRIVE release(DW 4.0 + DRIVE OS 5.2.6)?

I’m trying to setup a docker environment to test with a newer DRIVE release.

Up to this point we stick to the old DRIVE Software 10.

So I successfully set up an environment with DW 4.0 + DRIVE OS 5.2.6.

Even if the recording problem is solved, this will not help, because in our driving software we rely on the Nvidia DNNs which are not included in non Drive SW versions…

I will see how much effort it is to port the plugin nevertheless…

I now could test this in DW 4. Sadly with the same result.
I looked at the new examples and tried to follow and adapt my plugin to them.

One can see that in readRawData() the size of the whole message (canMessage + header) is 90 bytes:

Whereas validatePacket() gives me the correct 90 bytes, one can see that in pushData() the given size is 102 bytes:

Accordingly

[...]
const dwCANMessage* msg;
        if (!m_buffer.peek(reinterpret_cast<const uint8_t**>(&msg)))`
[...]

gives me garbage CAN messages…

The only message in the log is

[21-01-2022 07:53:22] RadarCustom::fillOutQueueHelper() decoding error.

but that can also be because of the parsing done wrong because of the garbage messages.
I am using this with the stock radar_replay_sample and have no idea where else to look for the error.

I would be very grateful for support, as we are now trying for a long time to get a CAN Radar plugin working.

EDIT: I Just tried the CAN Logger example with the can plugin example as they come out of the box. There, the size output in readRawData is also 90 bytes. But in pushData it is 78 bytes. So there its 12 bytes less, which made sense, if only the payload itself should be pushed.

EDIT2: Another thing I find strange: I cannot use

properties->packetSize = sizeof(dw::plugins::canbus::rawPacket);

instead, I used the following:

dwSensorPluginProperties props{};
props.packetSize = sizeof(dw::plugins::canbus::rawPacket);
properties = &props;

But I find it a bit strange, that in all other examples the pointer is not null. Is this special in the radar library?

Dear @f.petry,
Thank you for checking with DW 4.0. Could you share complete plugin code and used commands here. If can not be shared publicly, could please file a bug.

Please login to NVIDIA DRIVE - Autonomous Vehicle Development Platforms | NVIDIA Developer with your credentials. Please check MyAccount->MyBugs->Submit a new bug to file bug.
Please share ID here to follow up. Thanks

Thank you.

I filed a bug with ID 3513603.

Unfortunately, the format of the description got all messed up and I cannot edit it.

Dear @f.petry,
Thank you for sharing bug ID. We will follow up via bug.