[BUG] target-docker-container running driveworks sample_camera failed

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Required Info:

  • Software Version
    DRIVE OS 6.0.6
  • Target OS
    Linux
  • SDK Manager Version
    1.9.2.10884
  • Host Machine Version
    native Ubuntu Linux 20.04 Host installed with DRIVE OS DOCKER Containers

Describe the bug

following with this topic https://developer.nvidia.com/blog/running-docker-containers-directly-on-nvidia-drive-agx-orin/#entry-content-comments.
and this topic [BUG] target-docker-container running driveworks sample_hello_world failed

in target-docker-container, /usr/local/driveworks/bin/sample_camera failed with many errors.

To Reproduce

# start and into the container
./docker/run/orin_start.sh
./docker/run/orin_into.sh

the csv in /etc/nvidia-container-runtime/host-files-for-container.d/ is from here

the keypoint of orin_start.sh is

docker run --runtime nvidia --gpus all -it -d --privileged --name gw_orin_20.04_nvidia -e DOCKER_USER=nvidia -e USER=nvidia -e DOCKER_USER_ID=1000 -e DOCKER_GRP=nvidia -e DOCKER_GRP_ID=1000 -e DOCKER_IMG=arm64v8/ros:foxy -e USE_GPU=1 -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,graphics,video,utility,display -e NVIDIA_REQUIRE_JETPACK=csv-mounts=all -e DISPLAY=:1 -v /home/nvidia/zhensheng/orin_ws/nv_driveworks_demo/target:/target -v /dev:/dev -v /home/nvidia/zhensheng/cuda-sample:/home/nvidia/zhensheng/cuda-sample -v /home/nvidia/.cache:/home/nvidia/.cache -v /dev/bus/usb:/dev/bus/usb -v /media:/media -v /tmp/.X11-unix:/tmp/.X11-unix:rw -v /etc/localtime:/etc/localtime:ro -v /usr/src:/usr/src -v /lib/mgaules:/lib/mgaules --ulimit nofile=32768:32768 --ulimit msgqueue=2097152:2097152 -v /etc/nvsciipc.cfg:/etc/nvsciipc.cfg --net host --ipc host --cap-add SYS_ADMIN --cap-add SYS_PTRACE -w /target --add-host in_orin_docker:127.0.0.1 --add-host tegra-ubuntu:127.0.0.1 --hostname in_orin_docker --shm-size 2G -v /dev/null:/dev/raw1394 arm64v8/ros:foxy /bin/bash

the rig.json is edited and the camera with eligible module is connected.

Expected behavior

# in target-orin-host
/usr/local/driveworks/bin/sample_camera
ProgramArguments: Missing argument 'dwTracePath' requested
[21-05-2023 19:11:18] Platform: Detected Drive Orin P3710
[21-05-2023 19:11:18] TimeSource: monotonic epoch time offset is 1684599544108287
[21-05-2023 19:11:18] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[21-05-2023 19:11:18] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[21-05-2023 19:11:18] PTP Time is available from Eth Driver
[21-05-2023 19:11:18] Adding variable DW_Base:DW_Version
[21-05-2023 19:11:18] Added variable DW_Base:DW_Version
[21-05-2023 19:11:18] Platform: number of GPU devices detected 1
[21-05-2023 19:11:18] Platform: currently selected GPU device 0, Resource Data Dir: trt_08_05_10_03, Arch: ga10b
[21-05-2023 19:11:18] Platform: currently selected GPU device integrated ID 0
[21-05-2023 19:11:18] CUDLAEngine:getDLACount: CUDLA version is = 1003000
[21-05-2023 19:11:18] CUDLAEngine:getDLACount: Number of DLA devices = 2
[21-05-2023 19:11:18] Context::mountResourceCandidateDataPath resource FAILED to mount from './resources': VirtualFileSystem: Failed to mount './resources/resources.pak'
[21-05-2023 19:11:18] Context::mountResourceCandidateDataPath resource FAILED to mount from '/home/nvidia/zhensheng/orin_ws/nv_driveworks_demo/target/data': VirtualFileSystem: Failed to mount '/home/nvidia/zhensheng/orin_ws/nv_driveworks_demo/target/data/resources.pak'
[21-05-2023 19:11:18] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.10/bin/../data
[21-05-2023 19:11:18] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.10/bin/../data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.10/bin/../data/resources.pak'
[21-05-2023 19:11:18] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.10/data
[21-05-2023 19:11:18] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.10/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.10/data/resources.pak'
[21-05-2023 19:11:18] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/bin/../lib/libdw_base.so.5.10
[21-05-2023 19:11:18] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/targets/aarch64-Linux/lib/libdw_base.so.5.10
[21-05-2023 19:11:18] SDK: No resources(.pak) mounted, some modules will not function properly
[21-05-2023 19:11:18] SDK: use EGL display as provided
[21-05-2023 19:11:18] TimeSource: monotonic epoch time offset is 1684599544108287
[21-05-2023 19:11:18] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[21-05-2023 19:11:18] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[21-05-2023 19:11:18] PTP Time is available from Eth Driver
[21-05-2023 19:11:18] Initialize DriveWorks SDK v5.10.87
[21-05-2023 19:11:18] Release build with GNU 9.3.0 from buildbrain-branch-0-g9a5b4670e12 against Drive PDK v6.0.6.0
[21-05-2023 19:11:18] SensorFactory::createSensor() -> time.nvpps, nvpps-device=/dev/nvpps0
[21-05-2023 19:11:18] Can not create time sensor: DW_HAL_CANNOT_OPEN_CHANNEL: EndpointNVPPS: cannot open NVPPS device with name: /dev/nvpps0
 Error: Permission denied
[21-05-2023 19:11:18] fromFile: Loading rig file: /usr/local/driveworks-5.10/data/samples/sensors/camera/camera/rig.json
[21-05-2023 19:11:18] rig::DatabaseOverlayer: Failed to open a file '/tmp/car_vin' - file likely does not exist - skipping overlay
[21-05-2023 19:11:18] No valid data file found for camera:sample0 in parameter string: camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1 (using configuration folder /usr/local/driveworks-5.10/data/samples/sensors/camera/camera/)
[21-05-2023 19:11:18] rig::DatabaseOverlayer: Failed to open a file '/tmp/car_vin' - file likely does not exist - skipping overlay
onInitialize: creating camera.gmsl with params: camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[21-05-2023 19:11:18] SensorFactory::createSensor() -> camera.gmsl, camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[21-05-2023 19:11:18] CameraBase: pool size set to 8
[21-05-2023 19:11:18] SensorFactory::createSensor() -> camera.gmsl.master, 
[21-05-2023 19:11:18] CameraMaster::parseDevBlock Getting device info list.
[21-05-2023 19:11:18] devBlock: 0 Slave = 0 Interface = csi-ab Camera_name = V1SIM623S4RU5195NB3 Link = 0 GroupInit = 0 RecCfg = 1
[21-05-2023 19:11:18] Camera Match Name: V1SIM623S4RU5195NB3 Description: Sony IMX623 RGGB module - 120-deg FOV, MIPI-IMX623, MAX96717F linkIndex: 4294967295 serInfo.Name: MAX96717F
[21-05-2023 19:11:18] Client, Setting up information for camera ID 0
[21-05-2023 19:11:18] Client, successfully found info for camera ID 0 bound to id 0
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18]  ImageStreamer(NvMedia -> CUDA) 
[21-05-2023 19:11:18] CameraClient: using NITO found at /usr/share/camera/V1SIM623S4RU5195NB3.nito
[21-05-2023 19:11:18] SIPLMaster::SIPLMaster: Setting up master camera
[21-05-2023 19:11:18] Platform: 
[21-05-2023 19:11:18] Platform Config: 
[21-05-2023 19:11:18] Description: 
[21-05-2023 19:11:18] Number of device blocks: 1
[21-05-2023 19:11:18] Device Block : 0
[21-05-2023 19:11:18] 	csiPort: 2
[21-05-2023 19:11:18] 	i2cDevice: 0
[21-05-2023 19:11:18] 	Deserializer Name: MAX96712_Fusa_nv
[21-05-2023 19:11:18] 	Deserializer Description: Maxim 96712 Aggregator Fusa_nv
[21-05-2023 19:11:18] 	Deserializer i2cAddress: 41
[21-05-2023 19:11:18] 	Simulator Mode: 0
[21-05-2023 19:11:18] 	Passive Mode: 0
[21-05-2023 19:11:18] 	Phy Mode: 1
[21-05-2023 19:11:18] 	Number of camera modules: 1
[21-05-2023 19:11:18] 	CameraModule index: 0
[21-05-2023 19:11:18] 		Name :V1SIM623S4RU5195NB3
[21-05-2023 19:11:18] 		Description: Sony IMX623 RGGB module - 120-deg FOV, MIPI-IMX623, MAX96717F
[21-05-2023 19:11:18] 		Serializer name: MAX96717F
[21-05-2023 19:11:18] 		Serializer description: Maxim 96717F Serializer
[21-05-2023 19:11:18] 		Serializer i2cAdress: 64
[21-05-2023 19:11:18] 			EEPROM name: M24C04
[21-05-2023 19:11:18] 			EEPROM i2cAddress: 84
[21-05-2023 19:11:18] 			EEPROM description: M24C04 EEPROM
[21-05-2023 19:11:18] 			Sensor ID: 0
[21-05-2023 19:11:18] 			Sensor name: IMX623
[21-05-2023 19:11:18] 			Sensor description: Sony IMX623 Sensor
[21-05-2023 19:11:18] 			Sensor i2cAddress: 26
[21-05-2023 19:11:18] 			Sensor isTPGEnabled: 0
[21-05-2023 19:11:18] 			Sensor isTriggerMode: 1
[21-05-2023 19:11:18] 				 cfa: 33
[21-05-2023 19:11:18] 				 embeddedTopLines: 1
[21-05-2023 19:11:18] 				 embeddedBottomLines: 22
[21-05-2023 19:11:18] 				 inputFormat: 20
[21-05-2023 19:11:18] 				 height: 1536
[21-05-2023 19:11:18] 				 width: 1920
[21-05-2023 19:11:18] 				 fps: 30.000000
[21-05-2023 19:11:18] 				 Embedded Data: 0
[21-05-2023 19:11:18] CameraMaster available outputs: 1
[21-05-2023 19:11:18] CameraClient: setting pipeline config for camera ID 0
[21-05-2023 19:11:18] CameraMaster: master initiation
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
MAX96717F : Revision 4 detected!
Sensor IMX623 Rev 8 RGGB detected!
[21-05-2023 19:11:19] CameraMaster: Bootstrap NvSIPL and ClientPipelines complete
[21-05-2023 19:11:19] SAL: bootstrap complete
[21-05-2023 19:11:19] Initialize DriveWorks VisualizationSDK v5.10.87
[21-05-2023 19:11:19] Initialize DriveWorksGL SDK v5.10.87
onInitialize: Total cameras 1
onInitialize: getting image props 0
onInitialize: initilizing stream: 0
Main: Starting master.
[21-05-2023 19:11:19] CameraMaster: starting...
[21-05-2023 19:11:19] CameraClient: allocating image pool for ICP
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] Allocating NvSci group
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: allocate image pool for 0
[21-05-2023 19:11:19] Allocating NvSci ISP0 images
[21-05-2023 19:11:19] CameraClient: format not specified. Using h264 in Codec Header.
[21-05-2023 19:11:19] SIPLClient: Acquisition started
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:19] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] SIPLDeviceBlockNotificationHandler: Queue timeout
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
[21-05-2023 19:11:20] CameraClient: raw bit type is missing or unexpected in virtual channel info, meta info might be incomplete
Timing results:
Thread main:
-onProcess                CPU:     6us, std=   4       | GPU:     4us, std=   1       | samples=34
-onRender                 CPU: 33061us, std=9126       | GPU: 33084us, std=9119       | samples=34

[21-05-2023 19:11:21] CameraClient: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_CAPTURE_TIMEOUT
[21-05-2023 19:11:21] Sensor Health Signal for camera[0] camera.gmsl, camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
	timestamp: 1684696280493220	sourceID: 1040
	Sensor notifications: 
[21-05-2023 19:11:21] Sensor statistics for: camera[0] camera.gmsl, camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[21-05-2023 19:11:21] Statistics for DW_SENSOR_STATE_DELTA_HOST_AND_HOST_TIME
Events: 69
Errors: 1
Drops: 0
minDelta: 33332.0000 us
maxDelta: 33334.0000 us
meanDelta: 33332.7266us 
Standard deviation: 0.6166
[21-05-2023 19:11:21] Sensor statistics for: camera[0] camera.gmsl, camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[21-05-2023 19:11:21] Statistics for DW_SENSOR_STATE_DELTA_SENSOR_AND_SENSOR_TIME
Events: 69
Errors: 1
Drops: 0
minDelta: 0.0000 us
maxDelta: 0.0000 us
meanDelta: 0.0000us 
Standard deviation: 0.0000
[21-05-2023 19:11:21] Sensor statistics for: camera[0] camera.gmsl, camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[21-05-2023 19:11:21] Statistics for DW_SENSOR_STATE_DELTA_HOST_AND_SENSOR_TIME
Events: 69
Errors: 1
Drops: 0
minDelta: -2.8138e+14 us
maxDelta: -2.8138e+14 us
meanDelta: -2.8138e+14us 
Standard deviation: 0.0000
[21-05-2023 19:11:21] Sensor statistics for: camera[0] camera.gmsl, camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[21-05-2023 19:11:21] Statistics for DW_SENSOR_STATE_DELTA_CURRENT_AND_HOST_TIME
Events: 69
Errors: 1
Drops: 0
minDelta: 0.0000 us
maxDelta: 0.0000 us
meanDelta: 0.0000us 
Standard deviation: 0.0000
[21-05-2023 19:11:21]  Buffers(Processed) should be          released before calling DeInit() on sipl camera 
[21-05-2023 19:11:21] Deinit master camera
[21-05-2023 19:11:21] SIPLDeviceBlockNotificationHandler: Queue timeout
[21-05-2023 19:11:21] SIPLClient: ImageQueue timeout
[21-05-2023 19:11:21] SIPLClient: ImageQueue shutdown
[21-05-2023 19:11:21] SIPLNotifier: Event notification queue shutdown
[21-05-2023 19:11:22] SIPLDeviceBlockNotificationHandler: Queue timeout
[21-05-2023 19:11:23] SIPLDeviceBlockNotificationHandler: Queue timeout
[21-05-2023 19:11:24] CameraClient: Stopping client
[21-05-2023 19:11:24] Releasing camera master
[21-05-2023 19:11:24] SIPLDeviceBlockNotificationHandler: Queue timeout
[21-05-2023 19:11:24] Driveworks VisualizationSDK released
[21-05-2023 19:11:24] Releasing Driveworks SDK Context
[21-05-2023 19:11:24] DriveworksGL SDK released

Actual behavior

without installing systemd in the container

# in target-docker-container
/usr/local/driveworks/bin/sample_camera
ProgramArguments: Missing argument 'dwTracePath' requested
sh: 1: loginctl: not found
sh: 1: loginctl: not found
WindowGLFW: Failed create window
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)

after install systemd in the container

nvidia@in_orin_docker:/target$ /usr/local/driveworks/bin/sample_camera
ProgramArguments: Missing argument 'dwTracePath' requested
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to create bus connection: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to create bus connection: Host is down
WindowGLFW: Failed create window
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)

with docker run /lib/systemd/systemd tring to solve the System has not been booted issue. There still shows the error.

ProgramArguments: Missing argument 'dwTracePath' requested
WindowGLFW: Failed create window
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)

Additional context

  1. There is no sudo used. because it’s been solved by [BUG] target-docker-container running cuda-samples require unintended extra permission
3

with reading WindowGLFW: Failed create window - #6 by shayNV , I made some more tests

run sample with headless mode

# in target-orin-docker
nvidia@in_orin_docker:/usr/local/driveworks/bin$ ./sample_camera --offscreen=1
ProgramArguments: Missing argument 'dwTracePath' requested
WindowEGL: find EGL devices
WindowEGL: found 1 EGL devices
WindowEGL: init EGL with GLES3 context
WindowEGL: bind OpenGLES3 API
WindowEGL: offscreen mode -> use EGL PBuffer surface 1280x800
WindowEGL: create EGL context
WindowEGL: assign EGL context to current thread
[23-05-2023 07:27:53] Platform: Detected Drive Orin P3710
[23-05-2023 07:27:53] TimeSource: monotonic epoch time offset is 1684716041160715
[23-05-2023 07:27:53] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[23-05-2023 07:27:53] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[23-05-2023 07:27:53] PTP Time is available from Eth Driver
[23-05-2023 07:27:53] Adding variable DW_Base:DW_Version
[23-05-2023 07:27:53] Added variable DW_Base:DW_Version
[23-05-2023 07:27:53] Platform: number of GPU devices detected 1
[23-05-2023 07:27:53] Platform: currently selected GPU device 0, Resource Data Dir: trt_08_05_10_03, Arch: ga10b
[23-05-2023 07:27:53] Platform: currently selected GPU device integrated ID 0
[23-05-2023 07:27:53] CUDLAEngine:getDLACount: CUDLA version is = 1003000
[23-05-2023 07:27:53] CUDLAEngine:getDLACount: Number of DLA devices = 2
[23-05-2023 07:27:53] Context::mountResourceCandidateDataPath resource FAILED to mount from './resources': VirtualFileSystem: Failed to mount './resources/resources.pak'
[23-05-2023 07:27:53] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.10/bin/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.10/bin/data/resources.pak'
[23-05-2023 07:27:53] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.10/bin/../data
[23-05-2023 07:27:53] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.10/bin/../data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.10/bin/../data/resources.pak'
[23-05-2023 07:27:53] Context::findDataRootInPathWalk data/DATA_ROOT found at: /usr/local/driveworks-5.10/data
[23-05-2023 07:27:53] Context::mountResourceCandidateDataPath resource FAILED to mount from '/usr/local/driveworks-5.10/data': VirtualFileSystem: Failed to mount '/usr/local/driveworks-5.10/data/resources.pak'
[23-05-2023 07:27:53] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/bin/../lib/libdw_base.so.5.10
[23-05-2023 07:27:53] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/targets/aarch64-Linux/lib/libdw_base.so.5.10
[23-05-2023 07:27:53] SDK: No resources(.pak) mounted, some modules will not function properly
[23-05-2023 07:27:53] SDK: use EGL display as provided
[23-05-2023 07:27:53] TimeSource: monotonic epoch time offset is 1684716041160714
[23-05-2023 07:27:53] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[23-05-2023 07:27:53] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[23-05-2023 07:27:53] PTP Time is available from Eth Driver
[23-05-2023 07:27:53] Initialize DriveWorks SDK v5.10.87
[23-05-2023 07:27:53] Release build with GNU 9.3.0 from buildbrain-branch-0-g9a5b4670e12 against Drive PDK v6.0.6.0
[23-05-2023 07:27:53] SensorFactory::createSensor() -> time.nvpps, nvpps-device=/dev/nvpps0
[23-05-2023 07:27:53] Can not create time sensor: DW_HAL_CANNOT_OPEN_CHANNEL: EndpointNVPPS: cannot open NVPPS device with name: /dev/nvpps0
 Error: Permission denied
[23-05-2023 07:27:53] fromFile: Loading rig file: /usr/local/driveworks-5.10/data/samples/sensors/camera/camera/rig.json
[23-05-2023 07:27:53] rig::DatabaseOverlayer: Failed to open a file '/tmp/car_vin' - file likely does not exist - skipping overlay
[23-05-2023 07:27:53] No valid data file found for camera:sample0 in parameter string: camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1 (using configuration folder /usr/local/driveworks-5.10/data/samples/sensors/camera/camera/)
[23-05-2023 07:27:53] rig::DatabaseOverlayer: Failed to open a file '/tmp/car_vin' - file likely does not exist - skipping overlay
onInitialize: creating camera.gmsl with params: camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[23-05-2023 07:27:53] SensorFactory::createSensor() -> camera.gmsl, camera-name=V1SIM623S4RU5195NB3,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,async-record=1,file-buffer-size=16777216,skip-eeprom=1
[23-05-2023 07:27:53] CameraBase: pool size set to 8
[23-05-2023 07:27:53] SensorFactory::createSensor() -> camera.gmsl.master, 
libnvsipl_query.so: CNvMQuery.cpp : 108: GetDriverLibraries: Could not find module driver library directory, searched paths:
libnvsipl_query.so: CNvMQuery.cpp : 110: GetDriverLibraries:   - /usr/local/driveworks-5.10/bin/../../../../lib
libnvsipl_query.so: CNvMQuery.cpp : 110: GetDriverLibraries:   - /usr/lib/nvsipl_drv
libnvsipl_query.so: CNvMQuery.cpp : 110: GetDriverLibraries:   - /usr/local/cuda/lib64/nvsipl_drv
libnvsipl_query.so: CNvMQuery.cpp : 110: GetDriverLibraries:   - /opt/ros/foxy/opt/yaml_cpp_vendor/lib/nvsipl_drv
libnvsipl_query.so: CNvMQuery.cpp : 110: GetDriverLibraries:   - /opt/ros/foxy/lib/aarch64-linux-gnu/nvsipl_drv
libnvsipl_query.so: CNvMQuery.cpp : 110: GetDriverLibraries:   - /opt/ros/foxy/lib/nvsipl_drv
[23-05-2023 07:27:53] Driveworks exception thrown: DW_INTERNAL_ERROR: INvSIPLQuery::ParseDefault() failed! SIPL errror code = 10

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2023-05-23 07:27:53] DW Error DW_INTERNAL_ERROR executing DW function:
 dwSAL_createSensor(&m_camera[m_totalCameras], paramsClient[i], m_sal)
 at samples/sensors/camera/camera/main.cpp:343
Aborted (core dumped)

change DISPLAY

# in target-orin-docker
export DISPLAY=:0
/usr/local/driveworks/bin/sample_camera

ProgramArguments: Missing argument 'dwTracePath' requested
GLFW error: 65544, X11: Failed to open display :0
WindowGLFW: Failed to initialize GLFW
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)

ping @SivaRamaKrishnaNV

4

Dear @lizhensheng,
We have not evaluated DW samples on docker container. Could you use the target directly for unblocking your development?

5

Understood, the docker container in orin is just the fresh new feature.

But due to the advantage of using docker container, there is a strong need to run all dw samples in the docker container environment.

there are many reasons to use container. some of them is in following from Concepts — NVIDIA Cloud Native Technologies documentation

Containerizing GPU applications provides several benefits, among them:

  • Ease of deployment
  • Isolation of individual devices
  • Run across heterogeneous driver/toolkit environments
  • Requires only the NVIDIA driver to be installed on the host
  • Facilitate collaboration: reproducible builds, reproducible performance, reproducible results.

running the target directly is fine, but docker is the true need.

Thanks.

6

Hi, @SivaRamaKrishnaNV

With the effort of editing *.csv inject libs into docker container, the creatSensor issue was resolved.

/usr/local/driveworks/bin/sample_camera --offscreen=1 --rig=/usr/local/driveworks/data/samples/sensors/camera/camera/rig.json can be ran in the target docker container.

but the renderer issue still exists when offscreen=0 with correct display export DISPLAY=:1

/usr/local/driveworks/bin/sample_camera --offscreen=0 --rig=/usr/local/driveworks/data/samples/sensors/camera/camera/rig.json

output is

ProgramArguments: Missing argument 'dwTracePath' requested
GLFW error: 65542, EGL: Failed to get EGL display: Success
WindowGLFW: Failed create window
terminate called after throwing an instance of 'std::exception'
  what():  std::exception
Aborted (core dumped)
7

some times with this error which comes from cgf-app, but is believed to belong to dwCamera module.

<15>1 2023-05-29T06:05:25.217379Z - pilotPipe_process 2486 - - [1685340257827758us][VERBOSE][tid:0][ImageStreamerNvMediaCUDAImpl.hpp:73][NO_TAG]  ImageStreamer(NvMedia -> CUDA) 
<15>1 2023-05-29T06:05:25.223597Z - pilotPipe_process 2486 - - [1685340257833978us][VERBOSE][tid:0][ImageClasses.hpp:1099][ImageClasses] : handled exception. e.what= Error: importing external memory failed. Error cudaErrorUnknown: unknown error
 Allocation of NvSci Backing Store failed; skipping setup, dw format= 3001 pxlFmt= 6000, pxlType=64, planeCount=2 width=1920, height=1536, vidMem=0
<11>1 2023-05-29T06:05:25.228231Z - pilotPipe_process 2486 - - [1685340257838615us][ERROR][tid:0][Exception.hpp:328][NO_TAG] Driveworks exception thrown: Cannot allocate cuda pitch memory, bytes 2949120. Error cudaErrorUnknown: unknown error
<11>1 2023-05-29T06:05:25.228289Z - pilotPipe_process 2486 - - [1685340257838679us][ERROR][tid:0][Exception.hpp:328][NO_TAG] 
<11>1 2023-05-29T06:05:25.228502Z - pilotPipe_process 2486 - - [1685340257838892us][ERROR][tid:0][Runtime.cpp:247][Runtime] Failed to create node instance dwcgfRawCameraDeployPipe.cameraFrontLeftPipe.cameraCapture.cameraNode of type dw::framework::gwCameraNode. Details: DW_CUDA_ERROR: /gw_demo/modules/xplatform/xcgf/image_pipe/src/nodes/camera/camera/CameraNodeImpl.cpp:36 - DriveWorks Error

the error comes from the following code

8

Friendly ping @SivaRamaKrishnaNV

Could you help with the render issue?

9

Dear @lizhensheng,
I hope this issue is not a blocker for you.
As informed earlier, we plan to include DW samples testing in docker container for future release. Unfortunately, I don’t have an update from core team on this issue.

10

according to Display Issue :: X11 and GLFW in Driveworks Docker Environment Host PC - NGC GPU Cloud / Docker and NVIDIA Docker - NVIDIA Developer Forums

but xhost + before entering container not work in this case.

11

@SivaRamaKrishnaNV

Can you confirm this issue officially?

12

Hi @lizhensheng,
Do you mean which release the requested instructions available?

13

I mean, this bug should be reproduced by the nvidia forum.

Also, if you can provide the release that will solve the issue, I am happy to know the plan!

16

Friendly ping for any updates.