Hey!
I’m using the ROS2 Camera Helper Node to publish RGB images (as well as some bounding boxes). My main problem currently are the timestamps in the header of the image messages.
When I subscribe to the image-topic, I get this:
stamp:
sec: 1
nanosec: 500000078
frame_id: zone_1_cam_1_link
---
stamp:
sec: 1
nanosec: 533333413
frame_id: zone_1_cam_1_link
---
stamp:
sec: 1
nanosec: 533333413
frame_id: zone_1_cam_1_link
---
stamp:
sec: 1
nanosec: 566666748
frame_id: zone_1_cam_1_link
---
stamp:
sec: 1
nanosec: 566666748
frame_id: zone_1_cam_1_link
So multiple images share the same timestamp which of course creates all kind of problems in my pipeline later where I want to e.g. visualize the bounding boxes on the correct images. Without proper timestamps, I have no way to assign the correct detections to an image.
I even get the same effect when setting the useSystemTime-flag:
stamp:
sec: 1733394482
nanosec: 819141888
frame_id: zone_1_cam_1_link
---
stamp:
sec: 1733394482
nanosec: 819141888
frame_id: zone_1_cam_1_link
Any ideas on how I can fix that?
Hey @FooTheBar . What camera model are you using? I tried hawk cameras from nova_carter_navigation scene and it works well. Could you please provide a minimal setup of your USD scene so we can replicate the issue? Thanks!
Hey!
Thanks for your answer. I could reproduce the issue with an empty scene and the example-camera publisher:
repeating_timestamps.usda.txt (12.4 KB)
With the empty scene, only a small subset of images have the same timestamp, so I wrote a quick script to count the number of occurences.timestamp_check.py.txt (1.3 KB)
For the empty scene, I got 11 duplicated timestamps in 900 frames. But this value strongly depends on the complexity of the scene.
Since I got way more duplicates in my scene, I repeated the experiment in the large warehouse:
large_warehouse_single_cam.usda.txt (11.2 KB)
When looking into the empty part of the warehouse, I didn’t get a single repeated timestamp in 1000 frames, but when looking towards the shelves, I get 40 repeated timestamps in 1000 Frames, and in my normal scene with some people and moving forklifts up to 50%!.
I run locally on 22.04 with a RTX4080Super, 32GB RAM, i7-14700KF.
glpuga
February 21, 2025, 5:39pm
5
Just bumped into the same issue with 4.2.0.
No sorry. @zhengwang could you reproduce the issue?
1 Like
glpuga
February 24, 2025, 5:49pm
7
For the record, we tried updating the simulation to 4.5.0, and we still receive camera images with duplicated timestamps.
1 Like
Could this be related to my new post here?
hi @bruno.vetter , are you trying to control the timestamp on the ROS messages? The Read Simulation Time node can be useful for checking if there is any discrepancy between the published and simulation timestamp. Is that helpful?
What are your settings for the time steps per second for the usd Layer and for the physics scene?
1 Like
@FooTheBar sorry for my late replay. Thanks for your files. I am able to reproduce the issue. I am using the warehouse scene and let the perspective camera face the shelves. There is one repeating timestamp in ~2950 timestamps.
Thanks for replicating the issue! Which GPU/CPU did you use? Is your system far more powerful than mine?
I also have ubuntu 22.04.
^Czhengwang@zhengwang-isaac:~/isaacsim$ nvidia-smi
Fri Feb 28 22:20:48 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX 5000 Ada Gene... Off | 00000000:01:00.0 On | Off |
| 30% 48C P5 28W / 250W | 506MiB / 32760MiB | 15% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2707 G /usr/lib/xorg/Xorg 196MiB |
| 0 N/A N/A 2930 G /usr/bin/gnome-shell 173MiB |
| 0 N/A N/A 258479 G ...seed-version=20250212-050113.035000 63MiB |
+-----------------------------------------------------------------------------------------+
CUDA:
zhengwang@zhengwang-isaac:~/isaacsim$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
CPU:
zhengwang@zhengwang-isaac:~/isaacsim$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7950X 16-Core Processor
CPU family: 25
Model: 97
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 2
CPU max MHz: 5881.0000
CPU min MHz: 400.0000
BogoMIPS: 8982.95
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp
lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 mov
be popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topo
ext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsg
sbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl
xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoin
vd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2av
ic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_reco
v succor smca fsrm flush_l1d
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 512 KiB (16 instances)
L1i: 512 KiB (16 instances)
L2: 16 MiB (16 instances)
L3: 64 MiB (2 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Reg file data sampling: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; Safe RET
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Srbds: Not affected
Tsx async abort: Not affected
