DriveWorks FSYNC not enabled for trigger-mode GMSL cameras (DRIVE OS 7.0.3)

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
1

DRIVE OS Version: 7.0.3

Issue Description:

Summary

On DRIVE OS 7.0.3.0 (DriveWorks 7.0.3), nvsipl_camera captures two trigger-mode IMX728 GMSL cameras at a synchronized 30 fps, but a DriveWorks application (dwSensorCamera via a `camera.gmsl` rig) on the same hardware and rig runs at ~1 fps and logs Custom FSYNC is disabled. How do we make DriveWorks enable/program the FSYNC (TSC) group start so trigger-mode cameras run at full rate?

Environment

- Platform: DRIVE AGX Thor, board p3960-10-sw03, ENABLE_THOR_U=y

- Booted DTB: tegra264-p3960-0010-sw01-av-l-linux-gos (hypervisor / Guest-OS)

- DRIVE OS: 7.0.3.0

- DriveWorks: 7.0.3

- Cameras: 2x IMX728_RGGB (CPHY x4, trigger mode enabled, 30 fps)

- Deserializer / serializer: MAX96724_Fusa_nv / MAX96717F

- Topology: both cameras on one deserializer, CSI G/H, links 0 and 1

What works: nvsipl_camera reaches 30 fps synchronized

cd /opt/nvidia/drive-linux/samples/nvmedia/nvsipl/test/camera

./nvsipl_camera -c "IMX728_96724_RGGB_CPHY_x4" -m "0x0000 0x0000 0x0000 0x0011" --showfps

Result: both sensors report 29.999 fps, 0 drops. Verbose output shows SIPL programming the serializer FSYNC automatically:

CNvMTransportLink_Max96724_96717F: Trigger mode sync: true

SetSERFsync: Set FSYNC mode

What does not work: DriveWorks stays at ~1 fps

A DriveWorks app (sample_camera / our dwSensorCamera app) on the same rig logs:

Custom FSYNC is disabled

SIPLClient: Acquisition started

SIPLNotifier: Notification received ... NOTIF_WARN_ICP_FRAME_DROP

Both cameras then report ~0.94 fps (free-run fallback). Frames stream, but there is no FSYNC.

Device-tree state (verified on the running target)

We enabled fsync-groups in tegra264-p3960-camera.dtsi and reflashed the DTB. On the target:



/proc/device-tree/tsc_sig_gen@c230000/fsync-groups/status = okay

generator@380 / @480 / @500 : freq_hz = 30, status okay

generator@400 : freq_hz = 15

fsync-group@0 : generators <&gen0>, <&gen2>, <&gen3>   (30 Hz)

We understand from the DRIVE OS docs that once fsync-groups is enabled it is mandatory to program the start time (cam_fsync_program_abs_start_value). nvsipl_camera does this; DriveWorks apparently does not.

What we tried on the DriveWorks side

We added sync-gpio-idx-des=2 to the camera.gmsl rig parameter string. This had no effect: DriveWorks still logs Custom FSYNC is disabled and stays at ~1 fps (and no “invalid params” error). We could not find any rig parameter that enables custom FSYNC.

Questions

1. How does a DriveWorks dwSensorCamera / camera.gmsl rig enable custom FSYNC (program the TSC fsync-group start) on DRIVE OS 7.0.3? Is there a rig parameter token, a dwSensorCamera / SAL API call, an environment variable, or a required config? What are the exact valid values and format for sync-gpio-idx-des / sync-mux-sel-des on this board, and do they actually trigger the group start or only routing?

2. Is DriveWorks custom FSYNC gated on PTP sync? We see PTP sync check failed. Please confirm PTP sync establishment before running dwSal_start. This is a standalone unit with no PPS/GPS (nvpps in timer mode). If FSYNC is PTP-gated in DriveWorks, how do we enable synchronized capture without an external PPS/GPS source?

3. What is the recommended path to get synchronized, full-rate (30 fps) capture through DriveWorks for trigger-mode IMX728 on this platform, given that nvsipl_camera already achieves it on the same hardware?

Related observation

nvsipl_camera -b 0, and any re-init after a fsync group is already started, fails with Master initialization failed. status: 10. We believe this is the “group already started / no stop IOCTL” condition. Is rmmod/modprobe cam_fsync the intended reset, and is there a supported way to query whether a group is already started?

4

Dear @tarun_sathesh ,
Could you please share the complete nvsipl_camera command , application log with verbose mode, sample_camera log, rig.json with export DW_SIPL_NVMEDIA_VERBOSITY=4 is set in environment. Also, share syslog, dmesg messages corresponds to both application runs.

What changes were done in dtsi file and used commands to update dtb?

6

Sure @SivaRamaKrishnaNV , I am preparing the data now, will share them soon.

7

I am attaching the log files below. I shall add the remaining data shortly.
log_run_live_1fps.txt (1.8 KB)
log_run_live_verbose.txt (60.6 KB)
log_dmesg.txt (116.6 KB)
log_nvsipl_30fps.txt (2.0 KB)

8

Short version: with DW_SIPL_NVMEDIA_VERBOSITY=4, DriveWorks does configure the serializer for FSYNC (“PostSensorInit: Set FSYNC mode”, “Trigger mode sync: true”) - the same as nvsipl_camera - but it logs “Custom FSYNC is disabled” and never programs the fsync-group start, so the trigger-mode cameras free-run at ~0.94 fps. nvsipl_camera on the same rig reaches a synchronized 30 fps as soon as the group is started with -b 0.

Environment: Thor p3960-10-sw03, DRIVE OS 7.0.3.0, DriveWorks 7.0.3, 2x IMX728_RGGB on one MAX96724_Fusa_nv, CSI G/H, links 0 and 1.

1) nvsipl_camera - 30 fps (works)

Command:

./nvsipl_camera -c IMX728_96724_RGGB_CPHY_x4 -m "0x0000 0x0000 0x0000 0x0011" -b 0 --showfps

Result:

Sensor12  Frame rate (fps): 29.999   Frame captured: 300   Frame drops: 0
Sensor13  Frame rate (fps): 29.999   Frame captured: 299   Frame drops: 0

Note: it only reaches 30 fps with -b 0 (start fsync group 0). Without -b, on a fresh boot capture is 0 frames (trigger mode, group not started). Once started the group persists (no stop IOCTL); a second -b 0 returns “Master initialization failed. status: 10” until rmmod/modprobe cam_fsync.

2) run_live (our DriveWorks dwSensorCamera app) - ~0.94 fps (fails)

cam=0 fps=0.9375 ... 3840x2160
cam=1 fps=0.9375 ... 3840x2160

Both cameras run at ~0.94 fps (free-run fallback). This stays at ~0.94 fps even when the TSC group is already running (started beforehand via nvsipl_camera -b 0) - so a running generator alone does not help the DriveWorks path.

3) DriveWorks verbose (DW_SIPL_NVMEDIA_VERBOSITY=4)

From run_live (DriveWorks) - it configures the serializer FSYNC:

CNvMTransportLink_Max96724_96717F.cpp:142  DumpLinkParams: Trigger mode sync: true
CNvMTransportLink_Max96724_96717F.cpp:1246 SetSERFsyncLevel: set FSYNC pin to the low
CNvMTransportLink_Max96724_96717F.cpp:2178 PostSensorInit: Set Fsync
CNvMTransportLink_Max96724_96717F.cpp:2187 PostSensorInit: Set FSYNC mode

sample_camera (DriveWorks) additionally logs:

Custom FSYNC is disabled
SIPLClient: Acquisition started
SIPLNotifier: Notification received ... NOTIF_WARN_ICP_FRAME_DROP

So DriveWorks reaches “PostSensorInit: Set FSYNC mode” (same serializer setup as nvsipl), but prints “Custom FSYNC is disabled” and never programs the fsync-group start - which is exactly what nvsipl_camera’s -b does.

4) rig.json (thor_live.json)

{
  "rig": {
    "sensors": [
      {
        "name": "camera-name=imx728_cam1",
        "protocol": "camera.gmsl",
        "parameter": "camera-name=IMX728_RGGB,interface=csi-gh,link=0,deserializer=MAX96724_Fusa_nv,skip-eeprom=1,output-format=processed,CPHY-mode=1,enable-ultrafast-mode=1",
        "nominalSensor2Rig_FLU": { "quaternion": [0,0,0,1], "t": [0,0,0] },
        "properties": null
      },
      {
        "name": "camera-name=imx728_cam2",
        "protocol": "camera.gmsl",
        "parameter": "camera-name=IMX728_RGGB,interface=csi-gh,link=1,deserializer=MAX96724_Fusa_nv,skip-eeprom=1,output-format=processed,CPHY-mode=1,enable-ultrafast-mode=1",
        "nominalSensor2Rig_FLU": { "quaternion": [0,0,0,1], "t": [0,0,0] },
        "properties": null
      }
    ],
    "vehicle": { "valid": false },
    "vehicleio": []
  },
  "version": 8
}

5) Device-tree change and DTB update commands

File edited:

drive-foundation/platform-config/hardware/nvidia/platform/t264/automotive/kernel-dts/p3960/common/tegra264-p3960-camera.dtsi

Inside the tsc_sig_gen@c230000 node, the fsync-groups child ships with status = “disabled”. We changed it to “okay” (two occurrences - the ENABLE_PLATFORM_T2 branch and the default branch; we edited both). The generator@ nodes were already 30 Hz, so no other change.

Before:

tsc_sig_gen@c230000 {
    ...
    fsync-groups {
        status = "disabled";
        fsync-group@0 { id = <0>; status = "okay"; generators = <&gen0>, <&gen2>, <&gen3>; };
        fsync-group@1 { id = <1>; status = "okay"; generators = <&gen1>; };
    };
};

After:

tsc_sig_gen@c230000 {
    ...
    fsync-groups {
        status = "okay";
        fsync-group@0 { id = <0>; status = "okay"; generators = <&gen0>, <&gen2>, <&gen3>; };
        fsync-group@1 { id = <1>; status = "okay"; generators = <&gen1>; };
    };
};

Commands used to rebuild the DTB and flash (run inside the DRIVE OS build container):

cd drive-foundation
./make/bind_partitions -b p3960-10-sw03 drive_av.linux --storage_size 64g -p dev_nsr ENABLE_THOR_U=y
cd tools/flashtools/bootburn
./bootburn.py -b p3960-10-sw03 -x /dev/ttyACM2

Verified on the running target after flashing:

/proc/device-tree/tsc_sig_gen@c230000/fsync-groups/status = okay
generator@380 freq_hz=30, @480=30, @500=30, @400=15   (all status okay)
fsync-group@0 : generators <&gen0>, <&gen2>, <&gen3>

Refined questions

  1. DriveWorks already sets the serializer to trigger/FSYNC mode (“PostSensorInit: Set FSYNC mode”) but logs “Custom FSYNC is disabled” and does not start the fsync-group. How do we make DriveWorks (dwSensorCamera / camera.gmsl rig) program the fsync-group start - the equivalent of nvsipl_camera -b / cam_fsync_program_abs_start_value? Is there a rig parameter, a dwSensorCamera / SAL API, or an environment variable?

  2. Is the “Custom FSYNC is disabled” state gated on PTP? This is a standalone unit with no PPS/GPS (nvpps in timer mode; we also see “PTP sync check failed … Frame synchronization should not be expected”). If FSYNC is PTP-gated in DriveWorks, how do we enable it without an external PPS/GPS source?

  3. If DriveWorks 7.0.3 cannot program the fsync start itself, is nvsipl_camera the only supported path? If so, is there a supported way to start the group out-of-band and have a DriveWorks app consume the already-synchronized frames?

9

@SivaRamaKrishnaNV , if any more data is required from my side, please let me know.

10

@SivaRamaKrishnaNV , I have added a few more log files.

The command used for sample_camera is:

sudo /usr/local/driveworks/bin/sample_camera --rig /home/nvidia/Documents/thor_live.json --offscreen 2 2>&1 | tee /tmp/log_sample_camera_verbose.txt

log_sample_camera_verbose.txt (106.8 KB)
log_syslog.txt (249.2 KB)

11

@SivaRamaKrishnaNV , a temporary solution is to add the NVIDIA’s low-level cam_fsync C API [used by nvsipl_camera] to the wrapper code we made over DriveWorks. When tested, it is able to start FSYNC and we are getting 30 fps using DriveWorks. But there are a few questions regarding this approach, listed as follows:

  1. Is calling libcam_fsync (cam_fsync_program_abs_start_value / cam_fsync_stop_group) directly from a DriveWorks application the supported way to start FSYNC? Or does dwSensorCamera / the camera.gmsl rig have a native mechanism to program the fsync-group start that we should use instead?

  2. DriveWorks already does the serializer-side Set FSYNC mode but logs Custom FSYNC is disabled. Is there a rig parameter / config / env var to make DriveWorks itself program the group start? (sync-gpio-idx-des had no effect — what are its valid values on this board, and does it start the group or only route?)

  3. Ownership model: if an app calls cam_fsync directly while DriveWorks owns the cameras, who is responsible for start/stop, and how are multiple consumers / multiple apps handled? Is there a supported “start the group once at boot (system service), DriveWorks apps just consume” pattern?

  4. Safety/certification: the NvCamFsync.h header lists safety SGIDs/abilities — is direct application use of cam_fsync compatible with the safety story, or is it intended to be called only by SIPL internally?

  5. Reset/query: re-programming a running group returns CAM_FSYNC_GROUP_BUSY (status 1). Is cam_fsync_stop_group the intended clean reset, and is there a supported way to query whether a group is already started (to avoid the busy→stop→retry dance / rmmod modprobe cam_fsync)?

  6. On a standalone unit with no PPS/GPS (nvpps in timer mode), what is the supported way to get correct per-frame capture timestamps for time-ordered fusion? Is dwImageTimestamps::tscEofTimestampUs the recommended field?

  7. Units bug? tscEofTimestampUs is documented “in microseconds” but on our Thor it is raw TSC ticks at ~32 MHz (advances 1,066,667 per 30 Hz frame). Is that expected, or should it be µs? What is the canonical camera-TSC frequency for this platform so we don’t hard-code 32 MHz?

  8. Clock domains: cntfrq_el0 reads ~1 GHz while the camera-frame TSC is ~32 MHz. Are these expected to differ, and which clock is authoritative for cross-sensor (camera/CAN/IMU) time alignment?

13

Did you come to this conclusion based on the frame timestamp values from DW. If so, there is known issue in timestamp conversion in DW in DOS 7.0.3, which got fixed. So the camera is not running at 1FPS speed. On Thor clock frequency is 1GHz.

14

@SivaRamaKrishnaNV , thank you for your reply.

Yes — exactly. We reached the same conclusion: the ~0.94 fps was a display artifact from that timestamp conversion. Our app was computing fps from the per-frame timestamp, and with DW logging “Time sensor conversion of TSC to PTP failed, timestamp will be approximated” the approximated timestamps advance ~1 s per frame, so the number read ~1 fps regardless of the true rate. Measuring frame-count over wall-clock shows a steady 30 fps. Good to know it’s a known issue that’s fixed.

A few follow-ups so we can close this out:

1. Which release has the timestamp-conversion fix? Is it fixed in a specific DRIVE OS / DriveWorks version (e.g. 7.0.4+), or is there a patch/backport for 7.0.3? We’d like to move to the correct build.

2. The fps display was an artifact, but the underlying capture problem was real and separate. On a fresh boot our DriveWorks app (and sample_camera) captured 0 frames — DW does the serializer-side setup (“PostSensorInit: Set FSYNC mode”) but logs “Custom FSYNC is disabled” and never programs the TSC fsync-group start. We got a clean 30 fps only by calling libcam_fsync ourselves (cam_fsync_program_abs_start_value(group, tsc_now + 10 ms), cam_fsync_stop_group on shutdown) right after dwSAL_start — the same start that nvsipl_camera -b 0 does. Is calling libcam_fsync directly from a DriveWorks app the supported approach, or does dwSensorCamera / the camera.gmsl rig have a native option to program the group start that we should use instead?

3. Per-frame timestamps for fusion. Once we’re on the fixed build, do we get correct per-frame TSC timestamps on a standalone unit with no PPS/GPS (nvpps in timer mode, PTP sync not established)? We need correct capture timestamps for time-ordered fusion — is there a supported free-running/TSC timestamp mode that doesn’t require an external PPS/GPS source?

15

Dear @tarun_sathesh ,
Next devzone release has fix for timestamp conversion.
Could you please check fsync-1-offset-ns=<xxx>( rig parameter for nvsipl_camera -b / cam_fsync_program_abs_start_value). Let me know if it works?

16

@SivaRamaKrishnaNV ,

Thanks. We tested fsync-1-offset-ns two ways on DriveWorks 7.0.3 (Thor p3960-10-sw03), both without our manual libcam_fsync call:

  • Our dwSensorManager app, fsync-1-offset-ns=10000000 on the camera.gmsl rig → 0 frames.
  • NVIDIA sample_camera (single IMX728, --rig with fsync-1-offset-ns=10000000) → 0 frames: Custom FSYNC is disabled, then continuous NOTIF_WARN_ICP_CAPTURE_TIMEOUT and getNextFrame: DW_NOT_READY. Running sample_camera on the same rig without the parameter behaves identically — so the parameter has no effect in the DriveWorks camera.gmsl path (no parse error, no group start).

For reference, our app on the same hardware reaches a steady 30 fps as soon as we start the TSC group ourselves via cam_fsync_program_abs_start_value right after dwSAL_start — DriveWorks does the serializer setup (PostSensorInit: Set FSYNC mode) but never programs the group start.

So it looks like fsync-1-offset-ns is an nvsipl_camera parameter rather than a DriveWorks rig parameter. Could you confirm: (1) is there a DriveWorks camera.gmsl / dwSensorCamera mechanism that programs the fsync-group start, or is calling libcam_fsync directly the sanctioned approach on 7.0.3? (2) If fsync-1-offset-ns is meant to work through DriveWorks, what is the exact token/placement (per-sensor parameter vs. a separate field) and are additional settings required for dwSensorCamera to act on it?

I have attached the rig json file we have used (with the fsync-1-offset-ns parameter):

{
    "rig": {
        "sensors": [
            {
                "name": "camera-name=imx728_cam1",
                "protocol": "camera.gmsl",
                "parameter": "camera-name=IMX728_RGGB,interface=csi-gh,link=0,deserializer=MAX96724_Fusa_nv,skip-eeprom=1,output-format=processed,CPHY-mode=1,enable-ultrafast-mode=1,fsync-1-offset-ns=10000000",
                "nominalSensor2Rig_FLU": {
                    "quaternion": [0.0, 0.0, 0.0, 1.0],
                    "t": [0.0, 0.0, 0.0]
                },
                "properties": null
            },
            {
                "name": "camera-name=imx728_cam2",
                "protocol": "camera.gmsl",
                "parameter": "camera-name=IMX728_RGGB,interface=csi-gh,link=1,deserializer=MAX96724_Fusa_nv,skip-eeprom=1,output-format=processed,CPHY-mode=1,enable-ultrafast-mode=1,fsync-1-offset-ns=10000000",
                "nominalSensor2Rig_FLU": {
                    "quaternion": [0.0, 0.0, 0.0, 1.0],
                    "t": [0.0, 0.0, 0.0]
                },
                "properties": null
            }
        ],
        "vehicle": {
            "valid": false
        },
        "vehicleio": []
    },
    "version": 8
}
17

Dear @tarun_sathesh ,
Does fsync_debug helps your case with out additional changes in dtsi or calls in DW code? FYI, I have quickly verified changing frequency using fsync_debug at Can DriveWorks SDK camera sample change IMX728 frame rate from 30 FPS to 10 FPS? - #8 by SivaRamaKrishnaNV

Also, Could you please share the complete DW camera sample log.