DW_BUFFER_FULL on dwSAL_createSensor()

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.2.10884
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

We are getting a DW_BUFFER_FULL error when running the dwSAL_createSensor() on our unit. We are not sure why this would be happening and the doc’s for that function do not display that, that would be an error output to begin with.

Thank you for reaching out. To better understand the issue you’re facing, could you please provide us with the output of executing the ‘sample_camera’ application? This will help us gather more information about the error you’re encountering.

Additionally, could you confirm if you are experiencing the same issue with DriveWorks 5.10, which is compatible with DRIVE OS 6.0.6? This will help us determine if the issue is specific to the version you’re currently using.

We appreciate your cooperation and look forward to assisting you further.

Here is the output from the sample camera command:

I do notice in the argument for the sample camera it references “file-buffer-size” we do not set this in our command. Our camera is the IMX728 8m 3840x2160

Would this cause the DW_BUFFER_FULL error?

ProgramArguments: Missing argument 'dwTracePath' requested
[02-06-2023 12:58:44] Platform: Detected Drive Orin P3710
[02-06-2023 12:58:44] TimeSource: monotonic epoch time offset is 1685630224263624
[02-06-2023 12:58:44] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[02-06-2023 12:58:44] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[02-06-2023 12:58:44] PTP Time is available from Eth Driver
[02-06-2023 12:58:44] Adding variable DW_Base:DW_Version
[02-06-2023 12:58:44] Addded variable DW_Base:DW_Version
[02-06-2023 12:58:44] Platform: number of GPU devices detected 1
[02-06-2023 12:58:44] Platform: currently selected GPU device 0, Resource Data Dir: trt_08_04_11_06, Arch: ga10b
[02-06-2023 12:58:44] Platform: currently selected GPU device integrated ID 0
[02-06-2023 12:58:44] CUDLAEngine:getDLACountandCheck: CUDLA version is = 1002001
[02-06-2023 12:58:44] CUDLAEngine:getDLACountandCheck: Number of DLA devices = 2
[02-06-2023 12:58:44] ========== DLA Device 0 Attributes ==========
[02-06-2023 12:58:44]            UVA supported = 0
[02-06-2023 12:58:44]            Device version = 1
[02-06-2023 12:58:44]            Device 0 Check PASSED
[02-06-2023 12:58:44] ========== DLA Device 1 Attributes ==========
[02-06-2023 12:58:44]            UVA supported = 0
[02-06-2023 12:58:44]            Device version = 1
[02-06-2023 12:58:44]            Device 1 Check PASSED
[02-06-2023 12:58:44] Context::mountResourceCandidateDataPath resource FAILED to mount from './resources': VirtualFileSystem: Failed to mount './resources/resources.pak'
[02-06-2023 12:58:44] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.6/bin/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.6/bin/data/resources.pak'
[02-06-2023 12:58:44] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.6/data
[02-06-2023 12:58:44] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.6/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.6/data/resources.pak'
[02-06-2023 12:58:44] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.6/data
[02-06-2023 12:58:44] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.6/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.6/data/resources.pak'
[02-06-2023 12:58:44] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.6/targets/aarch64-Linux/lib/libdw_base.so.5.6
[02-06-2023 12:58:44] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.6/targets/aarch64-Linux/lib/libdw_base.so.5.6
[02-06-2023 12:58:44] SDK: No resources(.pak) mounted, some modules will not function properly
[02-06-2023 12:58:44] SDK: Create NvMediaDevice
[02-06-2023 12:58:44] SDK: use EGL display as provided
[02-06-2023 12:58:44] TimeSource: monotonic epoch time offset is 1685630224263624
[02-06-2023 12:58:44] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[02-06-2023 12:58:44] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[02-06-2023 12:58:44] PTP Time is available from Eth Driver
[02-06-2023 12:58:44] Initialize DriveWorks SDK v5.6.83
[02-06-2023 12:58:44] Release build with GNU 9.3.0 from buildbrain-branch-0-gdcc2cf22fff against Drive PDK v6.0.4.0
[02-06-2023 12:58:44] Rig::fromFile: Loading rig file: /usr/local/driveworks-5.6/data/samples/sensors/camera/camera/rig.json
[02-06-2023 12:58:44] rig::DatabaseOverlayer: Failed to open a file '/tmp/car_vin' - file likely does not exist - skipping overlay
[02-06-2023 12:58:44] No valid data file found for camera:sample0 in parameter string: camera-name=F008A120RM0A,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216 (using configuration folder /usr/local/driveworks-5.6/data/samples/sensors/camera/camera/)
[02-06-2023 12:58:44] rig::DatabaseOverlayer: Failed to open a file '/tmp/car_vin' - file likely does not exist - skipping overlay
onInitialize: creating camera.gmsl with params: camera-name=F008A120RM0A,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216
[02-06-2023 12:58:44] SensorFactory::createSensor() -> camera.gmsl, camera-name=F008A120RM0A,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216
[02-06-2023 12:58:44] Driveworks exception thrown: DW_FAILURE: CameraBase: The Camera module not ready on DriveWorks 5.6, see release notes. Will re-enable support ASAP in future release!

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2023-06-02 12:58:44] DW Error DW_FAILURE executing DW function:
 dwSAL_createSensor(&m_camera[m_totalCameras], paramsClient, m_sal)
 at /usr/local/driveworks/samples/src/sensors/camera/camera/main.cpp:311
Aborted
``

I couldn’t find any reference to ‘file-buffer-size’ in DriveWorks SDK Reference: Camera Sample.

Additionally, I noticed that there is no mention of the DW_BUFFER_FULL error in the output of your ‘sample_camera’ execution.

Can you provide more information on where you found the ‘file-buffer-size’ information and why the DW_BUFFER_FULL error is not present in the output?

In your sample command terminal output we see:

[02-06-2023 12:58:44] SensorFactory::createSensor() -> camera.gmsl, camera-name=F008A120RM0A,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216

I think file-buffer-size is probably for writing the output to a file, and not related to DW_BUFFER_FULL

As for why that output does not look like DW_BUFFER_FULL in sample_camera. I’ve attached two images in my original post. One is our code we have written and the terminal output with the DW_BUFFER_FULL error. The function giving us that error is dwSAL_createSensor()

Per the output of your ‘sample_camera’ execution, it encountered a different error, DW_FAILURE. You may want to check what’s different from your implementation.

Based on the configuration (HZKJ_IMX728_ES2-V2-120FOV_CPHY_x4), you are using a Quanta camera module. Per the table below (from DRIVE AGX Orin Sensors & Accessories | NVIDIA Developer), you should get Quanta’s support instead of us. However, I’ll check with our team if there is any document issue on dwSAL_createSensor().

I think i understand why it produced the DW_FAILURE code. I am also curious when seeing an output for status = dwSAL_createSensor() it’ll give us one of those enum codes DW_FAILURE or DW_BUFFER_FULL etc. however, is there a way to add more detail to the error output like we see in the sample application?

Which Quanta camera module are you using? HZKJ-1, HZKJ-2, or HZKJ-3? Have you been able to successfully run the sample_camera application with the camera module? Additionally, have you contacted Quanta regarding this issue? If so, what was their response?

Working with Quanta on the side and they are looking into this as well, As for the sample_camera with this specific module no we have not. Waiting to receive a Drive OS 6.0.6 driver from Quanta. If i try on the Driveworks 5.6 I assume i’ll get hit with this error again

[02-06-2023 12:58:44] Driveworks exception thrown: DW_FAILURE: CameraBase: The Camera module not ready on DriveWorks 5.6, see release notes. Will re-enable support ASAP in future release!

But we can try.

As for module its HZKJ-1

This forum is for the devkit (P3710) with DRIVE OS available on the site.
We kindly request you collaborate with Quanta regarding this issue. They will work with our team at their own discretion.

I’d still like to hear back about DW_BUFFER_FULL error in regards to dwSAL_createSensor() as that’s from your codebase and not Quantas implementation if that’s okay. It also saves us a step from going through multiple people!

I’ll let you know once any update.

1 Like

Just did a test with Drive OS 6.0.6 and Driveworks 5.10, we get the same error output.

When running ./nvsipl_camera in the /opt/nvidia/drive-linux/… it outputs that it’s reading the camera correctly with fps.

Cannot create sensor camera.gmsl with camera-name=HZKJ_IMX728_ES2_V2_120FOV_CPHY_x4,output-format=processed,interface=csi-a,link=0,CPHY-mode=1,isp2-res=3840X2160,nito-file=/usr/share/camera/HZKJ_IMX728_ES2_V2_120FOV.nito, error: DW_BUFFER_FULL

Ran the dwLogger and had this output

[06-06-2023 14:09:15] 
Platform: Detected Drive Orin P3710
[06-06-2023 14:09:15] 
TimeSource: monotonic epoch time offset is 1686055855372287
[06-06-2023 14:09:15] 
TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[06-06-2023 14:09:15] 
TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[06-06-2023 14:09:15] 
PTP Time is available from Eth Driver
[06-06-2023 14:09:15] 
Adding variable DW_Base:DW_Version
[06-06-2023 14:09:15] 
Added variable DW_Base:DW_Version
[06-06-2023 14:09:15] 
Platform: number of GPU devices detected 1
[06-06-2023 14:09:15] 
Platform: currently selected GPU device 0, Resource Data Dir: trt_08_05_10_03, Arch: ga10b
[06-06-2023 14:09:15] 
Platform: currently selected GPU device integrated ID 0
[06-06-2023 14:09:15] 
CUDLAEngine:getDLACount: CUDLA version is = 1003000
[06-06-2023 14:09:15] 
CUDLAEngine:getDLACount: Number of DLA devices = 2
[06-06-2023 14:09:15] 
Context::mountResourceCandidateDataPath resource FAILED to mount from './resources': VirtualFileSystem: Failed to mount './resources/resources.pak'
[06-06-2023 14:09:15] 
Context::mountResourceCandidateDataPath resource FAILED to mount from '/home/nvidia/data': VirtualFileSystem: Failed to mount '/home/nvidia/data/resources.pak'
[06-06-2023 14:09:15] 
Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.10/data
[06-06-2023 14:09:15] 
Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.10/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.10/data/resources.pak'
[06-06-2023 14:09:15] 
Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.10/data
[06-06-2023 14:09:15] 
Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.10/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.10/data/resources.pak'
[06-06-2023 14:09:15] 
Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/targets/aarch64-Linux/lib/libdw_base.so.5.10
[06-06-2023 14:09:15] 
Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/targets/aarch64-Linux/lib/libdw_base.so.5.10
[06-06-2023 14:09:15] 
SDK: No resources(.pak) mounted, some modules will not function properly
[06-06-2023 14:09:15] 
egl::Display: found 1 EGL devices
[06-06-2023 14:09:15] 
egl::Display: use drm device: drm-nvdc
[06-06-2023 14:09:15] 
TimeSource: monotonic epoch time offset is 1686055855372287
[06-06-2023 14:09:15] 
TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[06-06-2023 14:09:15] 
TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[06-06-2023 14:09:15] 
PTP Time is available from Eth Driver
[06-06-2023 14:09:15] 
Initialize DriveWorks SDK v5.10.87
[06-06-2023 14:09:15] 
Release build with GNU 9.3.0 from buildbrain-branch-0-g9a5b4670e12 against Drive PDK v6.0.6.0
[06-06-2023 14:09:15] 
SensorFactory::createSensor() -> time.nvpps, nvpps-device=/dev/nvpps0
[06-06-2023 14:09:15] 
Can not create time sensor: DW_HAL_CANNOT_OPEN_CHANNEL: EndpointNVPPS: cannot open NVPPS device with name: /dev/nvpps0
 Error: Permission denied
[06-06-2023 14:09:15] 
SensorFactory::createSensor() -> camera.gmsl, camera-name=HZKJ_IMX728_ES2_V2_120FOV_CPHY_x4,output-format=processed,interface=csi-a,link=0,CPHY-mode=1,isp2-res=3840X2160,nito-file=/usr/share/camera/HZKJ_IMX728_ES2_V2_120FOV.nito
[06-06-2023 14:09:15] 
Driveworks exception thrown: DW_BUFFER_FULL: The return string has insufficient capacity to hold the parsed param value
Cannot create sensor camera.gmsl with camera-name=HZKJ_IMX728_ES2_V2_120FOV_CPHY_x4,output-format=processed,interface=csi-a,link=0,CPHY-mode=1,isp2-res=3840X2160,nito-file=/usr/share/camera/HZKJ_IMX728_ES2_V2_120FOV.nito, error: DW_BUFFER_FULL
[06-06-2023 14:09:15] 
Releasing Driveworks SDK Context

The line that interests me the most is:

Driveworks exception thrown: DW_BUFFER_FULL: The return string has insufficient capacity to hold the parsed param value

You should reach out to Quanta to verify if the module is compatible with the devkit (P3710) (instead of P3363. please note). If it is compatible, please inquire about the DRIVE OS versions that support the module. They may need to provide drivers based on the supported versions.

In case you encounter difficulties in obtaining support from Quanta, kindly share their contact information with us. If you have any concerns about sharing the information publicly, feel free to send it via a personal message.

We have both SO / NITO drivers for Drive OS 6.0.6 and 6.0.4.

We are currently looking into getSensorParams and trying to see if we can glean anything from what’s on the system. I think the string we are passing is just incorrect, but i’m not sure how we would format the correct camera string

Thank you for providing the Quanta contact information. Could you please provide details regarding their support on this issue? It would be helpful to understand why Quanta has been unable to resolve the issue thus far. Additionally, please let me know how I can assist you further through the communication with them.

Hey Vick,

Some updates, we’ve gotten the BUFFER error to go away by specifying the correct interface. However, now when we run 3 cameras using the sample_camera application, passing in the rig file, the entire thing breaks. It works with 1 camera and 2 cameras, but not 3+. I’ve attached our syslog, consolelog and rig files to this post.
rig_a_3.json (3.0 KB)

quanta_syslog_06072023.txt (127.6 KB)
quanta_error_06072023.txt (28.0 KB)

Dear @eolson,
For 3 camera case, do you see issue when each camera connected to a seperate port?

When you mean port, like the 4 QUAD camera ports? A, B, C, D?

We’ve tried that and updated the rig file to be csi-a, csi-b, csi-c, however, in that instance it doesn’t seem to run. Not sure if i’m creating the rig correctly for that example. Let me test that out again