./recorder GMSL camera raw image buffer full error

Hi all,

I was following the instructions about how to set up cameras.
I changed config file recorder-config.json

"camera": {
        "separate-thread": true,
        "record-thread-priority": 0,
        "write-file-pattern": "video_*",
        "sensors": [
          {
                "protocol": "camera.gmsl",
                "params": "camera-type=ar0231-rccb,csi-port=ab,camera-count=1,output-format=raw,async-record=1,fifo-size=3",
                "channel-names": [
                                "first"
                             ]
          }
        ]
    },

and tried to run

sudo ./recorder CUDA_VISIBLE_DEVICES=0

, it kept showing the following error after several seconds of recording.

EncoderRaw: buffer is full, dropping data
CameraGMSLDriver: Serialization of image failed: DW_BUFFER_FULL
EncoderRaw: buffer is full, dropping data
CameraGMSLDriver: Serialization of image failed: DW_BUFFER_FULL
EncoderRaw: buffer is full, dropping data

However, when I changed into

output-format=h264

, it showed

CameraGMSL: serializer bitrate not specified. Using 8000000.

and the full buffer error disappeared.

Can anyone explain what went wrong?
Thanks.

Dear sf2584,
Can you check the issue with CUDA_VISIBLE_DEVICES=1 and let me know the PDK version if this does not fix.

Hi, SivaRamaKrishna,

The problem still exists when adding CUDA_VISIBLE_DEVICES=1. And I’m using the latest driveworks-0.6, DriveInstall 5.0.5.0b. For details, the whole process was printed:

sudo ./recorder CUDA_VISIBLE_DEVICES=1
Program Arguments:
--config-file=./recorder-config.json
--offscreen=0

nvrm_gpu: Bug 200215060 workaround enabled.
Initialize DriveWorks SDK v0.6.67
Release build with GNU 4.9.2 from v0.6.0-rc7-0-g51bd3aa against Vibrante PDK v5.0.5.0
TimeSource: monotonic epoch time offset is 1528315465946287
TimeSource: PTP ioctl returned error. Synchronized time will not be available.
TimeSource: Could not detect valid PTP time source at 'eth0'. Fallback to CLOCK_MONOTONIC.
SDK: Resources mounted from /usr/local/driveworks-0.6/data/resources
SDK: Create NvMediaDevice
SDK: use EGL display as provided
SDK: number of GPU devices detected 2
SDK: currently selected GPU device discrete ID 0
SDK: currently selected GPU device discrete ID 0
Recorder: Setting DriveWorks log level to verbose...
SAL: identified board as DrivePX2-TegraA
SensorFactory::createSensor() -> gps.uart, device=/dev/ttyACM0,baud=115200
Driveworks exception thrown: DW_HAL_CANNOT_OPEN_CHANNEL: EndpointTTY: cannot open device

SensorDriver: Cannot create sensor gps.uart device=/dev/ttyACM0,baud=115200
Cannot add gps sensors...
SensorFactory::createSensor() -> imu.xsens, device=0,frequency=100
Driveworks exception thrown: DW_HAL_CANNOT_OPEN_CHANNEL: EndpointXsens: Requested device 0 not found. Only 0 devices present

SensorDriver: Cannot create sensor imu.xsens device=0,frequency=100
Cannot add imu sensors...
SensorFactory::createSensor() -> can.socket, 
SensorCANSocket: expected device=can0 to be set to a valid can bus
Driveworks exception thrown: DW_INVALID_ARGUMENT: device params

SensorDriver: Cannot create sensor can.socket 
Cannot add can sensors...
SensorFactory::createSensor() -> lidar.socket, device=VELO_VLP16,ip=192.168.1.201,port=2368,scan-frequency=10
LidarSocket: no response from Lidar after 500ms
Driveworks exception thrown: DW_SAL_CANNOT_INITIALIZE: LidarSocket: not connected

SensorDriver: Cannot create sensor lidar.socket device=VELO_VLP16,ip=192.168.1.201,port=2368,scan-frequency=10
Cannot add lidar sensors...
SensorDriver: No support for recording from virtual sensors for radar.virtual
Cannot add radar sensors...
CameraGMSLDriver: output-format=raw switching output-format=raw+h264 for on screen camera preview. Only raw will be recorded.
SensorFactory::createSensor() -> camera.gmsl, camera-type=ar0231-rccb,csi-port=ab,camera-count=1,output-format=raw+yuv,async-record=1,fifo-size=3
CameraGMSL: required FPS = 30, resolution = 1920x1208
***** aurix state *****
 CAM_PWR_ON: 0x0
 TEGRA_GMSL: TEGRA B
 TEGRA_FPDL: TEGRA B
 TEGRA_A_HDMI: MXM
 TEGRA_B_HDMI: MXM
 TEGRA_PCI: TEGRA B
 FAN_STATE: UNKNOWN
***********************
***** aurix state *****
 CAM_PWR_ON: 0x1
 TEGRA_GMSL: UNKNOWN
 TEGRA_FPDL: UNKNOWN
 TEGRA_A_HDMI: UNKNOWN
 TEGRA_B_HDMI: UNKNOWN
 TEGRA_PCI: UNKNOWN
 FAN_STATE: UNKNOWN
***********************
Sensor AR0231 RCCB Rev7 detected!
nvmEglStProducerCreateCommon: Entered
NvMediaEglStreamWrapperInitializeEgl: Cannot find function NvEglApiGetAccess
nvmEglStProducerCreateCommon:Wrapper initialization failed
NvMediaEglStreamProducerCreate: Failed to Create Producer
StreamProducerNvMediaImage: could not create NvMedia Image stream producer!
Driveworks exception thrown: DW_CANNOT_CREATE_OBJECT: could not create NvMedia Image stream producer

Driveworks exception thrown: DW_INVALID_ARGUMENT: Cannot cast handle, given instance is a nullptr

CameraGMSLDriver: Cannot create NvMedia to CUDA streamer. Disabling render for camera 0.
Thread created for 1
CameraGMSLDriver: Could not read frame for channel 0 due to DW_NOT_READY
CameraGMSLDriver: Could not read frame for channel 0 due to DW_NOT_READY
Start recording...
EncoderRaw: buffer is full, dropping data
CameraGMSLDriver: Serialization of image failed: DW_BUFFER_FULL
EncoderRaw: buffer is full, dropping data
CameraGMSLDriver: Serialization of image failed: DW_BUFFER_FULL
EncoderRaw: buffer is full, dropping data
...

Dear sf2584,
Are you storing recording data on external SSD or PX2?

Hi,

I’m storing data directly on PX2.

Dear sf2584,
can you please share your config file and setup details to reproduce error our side. Also, please share Aurix version using sh /etc/systemd/scripts/nv_aurix_check_fw.sh

Hi SivaRamaKrishna,

The device is almost newly flashed with latest version and I’m testing one of the SEKONIX SF3325-10X cameras. I have posted the recorder-config.json setting in my first post.
For the result of using sh /etc/systemd/scripts/nv_aurix_check_fw.sh:

starting Aurix FW checking...
starting Aurix FW checking...
Checking arguments...
Rootfs EB FW version is 4.02.02.0
Response_From_Aurix
Flashed EB FW version is 4.02.02.00
/etc/systemd/scripts/nv_aurix_check_fw.sh: 283: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
/etc/systemd/scripts/nv_aurix_check_fw.sh: 284: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
/etc/systemd/scripts/nv_aurix_check_fw.sh: 285: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
/etc/systemd/scripts/nv_aurix_check_fw.sh: 286: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
Aurix is booted with Latest EB FW
Rootfs UPDATE FW version is 3.01.01.0
Flashed UPDATE FW version is 3.01.01.00
/etc/systemd/scripts/nv_aurix_check_fw.sh: 231: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
/etc/systemd/scripts/nv_aurix_check_fw.sh: 232: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
/etc/systemd/scripts/nv_aurix_check_fw.sh: 233: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
/etc/systemd/scripts/nv_aurix_check_fw.sh: 234: /etc/systemd/scripts/nv_aurix_check_fw.sh: [[: not found
Aurix is booted with Latest UPDATE FW

Dear sf2584,

Could you please help to check Aurix fw version via HostPC?

https://docs.nvidia.com/drive/nvvib_docs/index.html#page/NVIDIA%2520DRIVE%2520Linux%2520SDK%2520Development%2520Guide%2FFlashing%2FAURIX.html

Hi SteveNV,

It printed information as follows:

Info: Executing cmd: version, argc: 0, args: 
SW Version: DRIVE-V5.0.5-P2379-EB-Aurix-With3LSS-4.02.02
Compilation date: Nov  8 2017, 16:03:03
Command Executed

Dear sf2584,

Are you using USB Hub? I believe maybe not.
Could you please check the speed of the storage medium that is being recorded to (use dd if=/dev/zero of=<MOUNT_POINT>/testfile.txt bs=512k count=1024)? Thanks.

Dear sf2584,
We need complete json config file to investigate. Can you please share.

Hi SteveNV and SivaRamaKrishna,

No, I’m not using USB hub. I’ve check the speed, it shows:

nvidia@nvidia:~$ dd if=/dev/zero of=~/testfile.txt bs=512k count=1024
1024+0 records in
1024+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 4.50841 s, 119 MB/s

The json config file is shown as:

{
    "_comments": [
        "To record raw camera use output-format=raw.",
        "To record raw and h264 use output-format=raw+h264.",
        "To record h264 only use output-format=h264.",
        "To record lraw only use output-format=lraw.",
        "To record lraw and h264 use output-format=lraw+h264.",
        "To record camera asynchronously, add async-record=1 to ",
        "the params of the camera sensor.",
        "To record 2 consecutive frames and skip 6 consecutive frames in the ",
        "camera use frame-record-count=2 and frame-skip-count=6.",
        "The write-file-pattern for each only refers to the filenames ",
        "and not the extensions. Those are inferred automatically.",
        "The separate-thread for each refers to the fact of whether",
        "threads are shared or a new one is created for capturing data.",
        "If separate-thread=false, it will capture on the same thread as",
        "other sensor types that have set separate-thread=false.",
        "If separate-thread=true, a new thread will spawn just for that",
        "sensor type.",
        "To set the log-level choose from [verbose, debug, warn, error]"
        ],
    "version": "0.8",
    "gpu-device": 0,
    "path": ".",
    "log-level": "verbose",
    "file-buffer-size": 2097152,
    "camera": {
        "separate-thread": true,
        "record-thread-priority": 0,
        "write-file-pattern": "video_*",
        "sensors": [
          {
                "protocol": "camera.gmsl",
                "params": "camera-type=ar0231-rccb,csi-port=ab,camera-count=1,output-format=raw,async-record=1,fifo-size=3",
                "channel-names": [
                                "first"
                             ]
          }
        ]
    },

    "can" : {
        "separate-thread": true,
        "write-file-pattern": "can_*",
        "sensors" : [
            {
                "protocol": "can.socket",
                "params": "",
                "channel-name": "0"
            }
        ]
    },
    "gps" : {
        "separate-thread": false,
        "write-file-pattern": "gps_*",
        "lock-override": false,
        "sensors" : [
            {
              "protocol": "gps.uart",
              "params": "device=/dev/ttyACM0,baud=115200",
              "channel-name": "0"
            }
        ]
    },
    "imu" : {
        "separate-thread": false,
        "write-file-pattern": "imu_*",
        "sensors" : [
            {
              "protocol": "imu.xsens",
              "params": "device=0,frequency=100",
              "channel-name": "0"
            }
        ]
    },
    "lidar" : {
        "separate-thread": false,
        "write-file-pattern": "lidar_*",
        "sensors" : [
            {
                "protocol": "lidar.socket",
                "params": "device=VELO_VLP16,ip=192.168.1.201,port=2368,scan-frequency=10",
                "channel-name": "1"
            }
        ]
    },
    "radar" : {
        "write-file-pattern": "radar_*",
        "sensors" : [
            {
                "protocol": "radar.virtual",
                "params": "file=./radar_sample.bin",
                "channel-name": "0"
            }
        ]
    }
}

Thanks!

Dear sf2584,

Could you please re-try to capture after change async parameter value like “async-record=1” → “async-record=0” in the configuration file? Thanks.

Hey SteveNV,

It solves the problem. Thanks! Can you please tell me what’s the meaning of “async-record=0”?

Dear sf2584,

Could you please refer to page84 in the NVIDIA_DriveWorks_DevGuide.pdf for the option? Thanks.

  1. async-record - enables the camera to record asynchronously, on a separate thread than the camera was read
    from

Thanks so much!