DRIVE OS Version: 7.0.3
Issue Description:
Summary
On DRIVE OS 7.0.3 / DriveWorks 7.0.3, a process that streams GMSL cameras via SensorManager and then shuts down intermittently SIGBUSes inside dwSAL_release(), in FrameAccumulator::~FrameAccumulator (libdw_sensors_std.so.7.0), jumping through a corrupted vtable pointer.
It is teardown-only — capture is unaffected and the fault does not poison the next run — but it happens on a meaningful fraction of clean shutdowns (~8–17% in our application) and we would like a fix or a documented correct teardown sequence.
We reduced it to a minimal reproducer that uses public SensorManager API only (no CUDA, no image APIs, no GL, no framework) which faults at ~100% on our rig. The full source, build file, and rig are inline at the bottom of this post.
Environment
- DRIVE AGX Thor (p3960-10-sw03), DRIVE OS 7.0.3.0, DriveWorks 7.0.3
- 2× IMX728 GMSL trigger-mode cameras, one MAX96724_Fusa_nv deserializer on
csi-gh - Cameras added as camera-group siblings via
dwSensorManager_addCameraSensor, withdwSAL_start()beforedwSensorManager_start() - Trigger-mode FSYNC WAR (
cam_fsync_program_abs_start_value) from this thread is in use, because without it the cameras produce no frames.
Backtrace
Captured with gdb on the target:
Thread 1 received signal SIGBUS, Bus error.
#0 0x00000000000000?? in ?? () <- jump through a corrupted vtable pointer (value varies per run: 0x31, 0x61, ...)
#1 dw::sensors::camera::CameraClient::SIPLClient::FrameAccumulator::~FrameAccumulator()
from /usr/local/driveworks/targets/aarch64-Linux/lib/libdw_sensors_std.so.7.0
#3 dw::sensors::camera::CameraClient::SIPLClient::~SIPLClient()
#4 dw::sensors::camera::CameraClient::deinitClient()
#5 dw::sensors::camera::CameraClient::~CameraClient()
#8 dw::sensors::SensorFactory::releaseSensor(dw::sensors::Sensor const*)
#9 dw::sensors::SensorFactory::~SensorFactory()
#12 dwSAL_release() from libdw_sensors_std.so.7.0
#13 main()
#0 is a jump to a low garbage address — a call through a corrupted vtable pointer, so the FrameAccumulator is already destroyed or overwritten by the time its destructor runs.
Thread state at the moment of the fault — two SIPL worker threads and the NVPPS endpoint thread are still alive while the main thread destroys the camera client:
Thread 16 __futex_abstimed_wait_common64 ... pthread_cond_wait <-- SIPL worker, alive
Thread 5 __futex_abstimed_wait_common64 ... pthread_cond_wait <-- SIPL worker, alive
Thread 4 "dw::hal_endpnt" select() <- EndpointNVPPS::waitForEvent <-- alive
Thread 3 "cuda-EvtHandlr" poll() (libcuda)
Thread 2 "cuda0000180000" poll() (libcuda)
Thread 1 <-- crashing in FrameAccumulator::~FrameAccumulator
This looks like a teardown race in which the SIPL workers are not joined before the objects they reference (FrameAccumulator) are destroyed.
How to reproduce
Build (native on the DRIVE AGX target with DriveWorks installed) and run in a loop:
cmake -B build -DCMAKE_BUILD_TYPE=Release .
cmake --build build
for i in $(seq 1 40); do
./build/repro_sal_release_crash --rig=rig.json --seconds=6
rc=$?; [ $rc -ge 128 ] && echo "run $i FAULT rc=$rc (signal $((rc-128)))"
done
The program does only this: dwInitialize → dwSAL_initialize → dwRig_initializeFromFile → dwSensorManager_initialize → add the cameras as siblings → dwSAL_start → dwSensorManager_start → start FSYNC → stream events for --seconds → on SIGINT (raised by a watchdog thread, simulating Ctrl-C), tear down (releaseAcquiredEvent → dwSensorManager_stop → dwSensorManager_release → dwRig_release → dwSAL_release → dwRelease).
Each run prints its frame count and clean exit, or dies with Bus error (core dumped) (rc 135) inside dwSAL_release().
The isolated trigger
We bisected the reproducer (each figure ≥ 40 runs):
| Configuration | Faults |
|---|---|
| Stream, release each event during the loop, hold nothing at teardown | 0 / 140 |
Stream, decode each frame (dwSensorCamera_getImage / dwImage_getCUDA + a CUDA copy), still hold nothing |
0 / 40 |
| Stream, hold the last acquired event across teardown, release it as the first teardown step — with decode | 39 / 40 |
| Same, without decode (pure SensorManager) | 40 / 40 |
So the crash requires exactly two things beyond a normal session:
- frames must have flowed (hence the FSYNC WAR — a session that streams nothing tears down cleanly every time, 0/24), and
- an acquired event must still be held when teardown begins, and be released (
dwSensorManager_releaseAcquiredEvent) as the first teardown step, beforedwSensorManager_stop(), while the SIPL workers are still live.
Frame decode is not required. Releasing an outstanding event at teardown-start — while the worker threads above are still parked — is what provokes the vtable corruption.
What we tried
Reordering teardown to call dwSensorManager_stop() before releasing the held event (to quiesce the workers first) looked promising on the reproducer but did not significantly reduce the fault rate in our real application (release-first 8/100 vs stop-first 4/100 over a 100-pair A/B, not significant), and releasing an event after stop() is of unclear safety. So we do not have a reliable application-side workaround.
Questions
- Is it supported to hold an acquired event (
dwSensorManager_acquireNextEvent) and release it during teardown, and is there a required ordering ofdwSensorManager_releaseAcquiredEvent/dwSensorManager_stop/dwSensorManager_release/dwSAL_releasethat we are violating? - Should
dwSensorManager_stop()/dwSensorManager_release()join the SIPL worker threads before the camera client and itsFrameAccumulatorare destroyed? - Is
dwSAL_release()expected to release camera sensors thatdwSensorManager_release()has already released? The backtrace shows the camera client being destroyed from insidedwSAL_release(), afterdwSensorManager_release()returned — a double-release would explain the corrupted vtable. - Is there a required teardown step we are missing (e.g. a per-sensor release, or a counterpart to
dwSAL_start()— there is nodwSAL_stop())?
Happy to provide gdb sessions, core dumps, or additional logs.
Full reproducer source (repro_sal_release_crash.cpp)
// Minimal reproducer: SIGBUS inside dwSAL_release() at teardown on DRIVE OS 7.0.3.
//
// Public DriveWorks SensorManager API only — no CUDA, no image APIs, no application
// framework, no GL. Run:
//
// ./repro_sal_release_crash --rig=rig.json --seconds=6
//
// It streams cameras for `--seconds`, then a watchdog raises SIGINT (as if the operator
// pressed Ctrl-C), and it tears down and exits. Run it in a loop; on our rig it SIGBUSes
// inside dwSAL_release() on the majority of iterations.
//
// THE TRIGGER (isolated by bisection):
// 1. frames must actually have flowed, and
// 2. an acquired SensorManager event must still be HELD when teardown begins, and
// released (dwSensorManager_releaseAcquiredEvent) as the first teardown step,
// BEFORE dwSensorManager_stop(), while the SIPL worker threads are still live.
// A variant that releases every event during streaming and holds nothing at teardown
// does NOT crash (0/140). Decoding the frames is NOT required.
//
// NOTE ON FSYNC: these are trigger-mode GMSL cameras, and DriveWorks does not start the
// TSC generator group ("Custom FSYNC is disabled"). We therefore call
// cam_fsync_program_abs_start_value() after dwSensorManager_start() (the WAR from
// https://forums.developer.nvidia.com/t/375199 ). Without frames flowing the crash never
// occurs, so the WAR is required to reproduce; it is not itself implicated.
#include <atomic>
#include <chrono>
#include <csignal>
#include <cstdint>
#include <cstdio>
#include <dlfcn.h>
#include <signal.h>
#include <string>
#include <thread>
#include <unistd.h>
#include <unordered_map>
#include <dw/core/base/Status.h>
#include <dw/core/base/VersionCurrent.h>
#include <dw/core/context/Context.h>
#include <dw/rig/Rig.h>
#include <dw/sensors/common/SensorTypes.h>
#include <dw/sensors/common/Sensors.h>
#include <dw/sensors/sensormanager/SensorManager.h>
#define CHECK(x) \
do { \
dwStatus s__ = (x); \
if (s__ != DW_SUCCESS) { \
std::printf("FAILED %s: %s\n", #x, dwGetStatusName(s__)); \
return 1; \
} \
} while (0)
namespace {
// Teardown is triggered by an asynchronous SIGINT arriving mid-stream, like an operator
// pressing Ctrl-C. A clean, drained exit does not reproduce as reliably; the signal
// interrupts the SIPL worker threads mid-flight, which the backtrace shows are still live
// at the moment of the fault.
std::atomic<bool> g_running{true};
extern "C" void handle_signal(int /*sig*/) { g_running.store(false); }
// --- the cam_fsync WAR (trigger-mode GMSL cameras need the TSC pulse to produce frames) ---
std::uint64_t tsc_now_ticks() {
std::uint64_t t = 0;
__asm__ __volatile__("isb; mrs %0, cntvct_el0; isb" : "=r"(t));
return t;
}
std::uint64_t cntvct_hz() {
std::uint64_t hz = 0;
__asm__ __volatile__("mrs %0, cntfrq_el0" : "=r"(hz));
return hz;
}
void* g_fsync_lib = nullptr;
int (*g_fsync_stop)(std::uint32_t) = nullptr;
bool start_fsync(std::uint32_t group) {
g_fsync_lib = dlopen("libcam_fsync.so", RTLD_NOW | RTLD_GLOBAL);
if (g_fsync_lib == nullptr) {
std::printf("dlopen libcam_fsync.so: %s\n", dlerror());
return false;
}
auto program = reinterpret_cast<int (*)(std::uint32_t, std::uint64_t)>(
dlsym(g_fsync_lib, "cam_fsync_program_abs_start_value"));
g_fsync_stop = reinterpret_cast<int (*)(std::uint32_t)>(dlsym(g_fsync_lib, "cam_fsync_stop_group"));
if (program == nullptr || g_fsync_stop == nullptr) {
return false;
}
const std::uint64_t guard = cntvct_hz() / 100;
int rc = program(group, tsc_now_ticks() + guard);
if (rc == 1) { // GROUP_BUSY: left running by a prior run
g_fsync_stop(group);
rc = program(group, tsc_now_ticks() + guard);
}
return rc == 0;
}
// Cameras behind one deserializer must be added as SIBLINGS of one camera-group, or
// dwSensorManager_start() fails with DW_INTERNAL_ERROR on the second camera. Group by
// the parameter string's `interface=` value.
std::string camera_group_of(const char8_t* parameter, std::uint32_t id) {
const std::string params{parameter != nullptr ? reinterpret_cast<const char*>(parameter) : ""};
const std::string key{"interface="};
const auto pos = params.find(key);
if (pos == std::string::npos) {
return "cam-" + std::to_string(id);
}
const auto start = pos + key.size();
const auto end = params.find(',', start);
return params.substr(start, end - start);
}
std::string arg_value(int argc, char** argv, const std::string& name, const std::string& fallback) {
const std::string prefix = "--" + name + "=";
for (int i = 1; i < argc; ++i) {
const std::string a{argv[i]};
if (a.rfind(prefix, 0) == 0) {
return a.substr(prefix.size());
}
}
return fallback;
}
} // namespace
int main(int argc, char** argv) {
const std::string rig = arg_value(argc, argv, "rig", "rig.json");
const int seconds = std::stoi(arg_value(argc, argv, "seconds", "6"));
dwContextHandle_t ctx = DW_NULL_HANDLE;
dwSALHandle_t sal = DW_NULL_HANDLE;
dwRigHandle_t rig_handle = DW_NULL_HANDLE;
dwSensorManagerHandle_t sm = DW_NULL_HANDLE;
dwContextParameters ctx_params{};
CHECK(dwInitialize(&ctx, DW_VERSION, &ctx_params));
CHECK(dwSAL_initialize(&sal, ctx));
CHECK(dwRig_initializeFromFile(&rig_handle, ctx, rig.c_str()));
CHECK(dwSensorManager_initialize(&sm, 16, sal));
std::uint32_t sensor_count = 0;
CHECK(dwRig_getSensorCount(&sensor_count, rig_handle));
std::unordered_map<std::string, std::uint32_t> next_sibling;
for (std::uint32_t id = 0; id < sensor_count; ++id) {
const char8_t* protocol = nullptr;
const char8_t* parameter = nullptr;
CHECK(dwRig_getSensorProtocol(&protocol, id, rig_handle));
CHECK(dwRig_getSensorParameterUpdatedPath(¶meter, id, rig_handle));
dwSensorParams params{};
params.protocol = protocol;
params.parameters = parameter;
if (std::string{reinterpret_cast<const char*>(protocol)}.rfind("camera", 0) == 0) {
const std::string group = camera_group_of(parameter, id);
CHECK(dwSensorManager_addCameraSensor(group.c_str(), next_sibling[group]++, params, id, sm));
} else {
CHECK(dwSensorManager_addSensor(params, id, sm));
}
}
// dwSAL_start BEFORE dwSensorManager_start: builds the one shared CameraMaster up
// front, so the per-sensor starts attach to it instead of colliding.
CHECK(dwSAL_start(sal));
CHECK(dwSensorManager_start(sm));
// FSYNC only works if started AFTER the sensors are up (see note at the top).
if (!start_fsync(0)) {
std::printf("WARNING: cam_fsync WAR failed; cameras will not stream and the crash "
"will NOT reproduce\n");
}
// A watchdog raises SIGINT after `seconds`, so the teardown is triggered by an async
// signal (like Ctrl-C) rather than a drained, clean exit.
std::signal(SIGINT, handle_signal);
std::signal(SIGTERM, handle_signal);
std::thread watchdog{[seconds]() {
std::this_thread::sleep_for(std::chrono::seconds(seconds));
kill(getpid(), SIGINT); // process-directed, like `kill -INT <pid>`
}};
// Stream, HOLDING the last acquired event across teardown. We release the previous
// event only when we acquire the next, so at the SIGINT one event is still held; it is
// released as the first teardown step below. Holding an event across teardown is the
// condition that provokes the crash. The steady_clock check is only a hang guard in
// case the signal is missed.
std::uint64_t frames = 0;
const dwSensorEvent* held = nullptr;
const auto fallback = std::chrono::steady_clock::now() + std::chrono::seconds(seconds + 5);
while (g_running.load() && std::chrono::steady_clock::now() < fallback) {
const dwSensorEvent* ev = nullptr;
const dwStatus s = dwSensorManager_acquireNextEvent(&ev, 100000, sm);
if (s == DW_TIME_OUT || s == DW_END_OF_STREAM) {
continue; // keep the currently held event
}
if (s != DW_SUCCESS) {
std::printf("acquireNextEvent: %s\n", dwGetStatusName(s));
break;
}
// Release the previously held event (drained), keep the new one held.
if (held != nullptr) {
dwSensorManager_releaseAcquiredEvent(held, sm);
}
held = ev;
if (ev->type == DW_SENSOR_CAMERA) {
frames += ev->numCamFrames;
}
}
watchdog.join();
std::printf("streamed %lu camera frames (stopped by %s)\n", static_cast<unsigned long>(frames),
g_running.load() ? "fallback deadline" : "SIGINT");
// ---- teardown ----
// Release the still-held event FIRST, then stop — this is the ordering that faults.
// (Reversing these two, stop() before the release, was tried as a workaround and does
// NOT reliably help on the real application, so this reproducer keeps the natural
// order.) The fault lands a few calls later, inside dwSAL_release().
if (held != nullptr) {
dwSensorManager_releaseAcquiredEvent(held, sm);
held = nullptr;
}
dwSensorManager_stop(sm);
if (g_fsync_stop != nullptr) {
g_fsync_stop(0);
}
dwSensorManager_release(sm);
dwRig_release(rig_handle);
dwSAL_release(sal); // <-- SIGBUS here, inside
// CameraClient::~CameraClient -> FrameAccumulator::~FrameAccumulator
dwRelease(ctx);
std::printf("clean exit\n");
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.10)
project(repro_sal_release_crash CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(DRIVEWORKS_DIR "/usr/local/driveworks" CACHE PATH "DriveWorks SDK root")
find_library(DW_SENSORS dw_sensors_std PATHS "${DRIVEWORKS_DIR}/lib" REQUIRED)
find_library(DW_BASE dw_base PATHS "${DRIVEWORKS_DIR}/lib" REQUIRED)
find_package(Threads REQUIRED)
add_executable(repro_sal_release_crash repro_sal_release_crash.cpp)
target_include_directories(repro_sal_release_crash PRIVATE "${DRIVEWORKS_DIR}/include")
target_link_libraries(repro_sal_release_crash
PRIVATE ${DW_SENSORS} ${DW_BASE} ${CMAKE_DL_LIBS} Threads::Threads
)
rig.json (2× IMX728 GMSL trigger-mode behind one MAX96724)
{
"rig": {
"sensors": [
{
"name": "imx728_cam1",
"protocol": "camera.gmsl",
"parameter": "camera-name=IMX728_RGGB,interface=csi-gh,link=0,deserializer=MAX96724_Fusa_nv,skip-eeprom=1,output-format=processed,CPHY-mode=1,enable-ultrafast-mode=1",
"nominalSensor2Rig_FLU": {
"quaternion": [0.0, 0.0, 0.0, 1.0],
"t": [0.0, 0.0, 0.0]
},
"properties": null
},
{
"name": "imx728_cam2",
"protocol": "camera.gmsl",
"parameter": "camera-name=IMX728_RGGB,interface=csi-gh,link=1,deserializer=MAX96724_Fusa_nv,skip-eeprom=1,output-format=processed,CPHY-mode=1,enable-ultrafast-mode=1",
"nominalSensor2Rig_FLU": {
"quaternion": [0.0, 0.0, 0.0, 1.0],
"t": [0.0, 0.0, 0.0]
},
"properties": null
}
],
"vehicle": {
"valid": false
},
"vehicleio": []
},
"version": 8
}