Environment
- Jetson AGX Xavier (Connect Tech AGX101 Rogue carrier)
- L4T R35.6.5 (
nvidia-l4t-core/nvidia-l4t-multimedia/nvidia-l4t-jetson-multimedia-api35.6.5-20260701022834) libnvjpeg.somd51442835f5142724df475ac8502d48f40- The same application on L4T R32.6.1 has never shown this crash.
Summary
On R35.6 we get intermittent SIGSEGVs inside hardware JPEG encoding via the stock NvJPEGEncoder::encodeFromFd() sample class. The encode is a 4096×2160 YUV420 NvBufSurface at quality 99, run at about 1 Hz from one thread of a multi-threaded process.
Minidumps and disassembly point to a bug in libnvjpeg itself:
jpegTegraEncoderMgrCreateregisters the NvMM block’s event callback with the address of one of its own stack locals as the client context, not the manager object.- The callback treats that context as the manager object.
- On event type 1, it signals two semaphores it reads at
ctx+2328andctx+2336. OncejpegTegraEncoderMgrCreatehas returned,ctxpoints into a dead stack frame. Those “semaphores” are whatever happens to be on the stack, andNvOsSemaphoreSignal()→pthread_mutex_lock()faults.
Event type 1 appears to be the block’s error notification. libnvmmlite_image raises it when the hardware operation started by command 1 returns a status other than 0 or 14. In both of our crash dumps that status was 4. So the sequence was: a hardware JPEG operation failed, and the error-reporting path crashed instead of returning the error.
Disassembly (libnvjpeg.so, R35.6.5)
Registration in jpegTegraEncoderMgrCreate:
3c14c: f9402fe0 ldr x0, [sp, #88] // x0 = mgr (local variable)
3c150: b00000e2 adrp x2, 0x59000
3c154: 910163e1 add x1, sp, #88 // x1 = &mgr <-- address of the stack local
3c158: f947b842 ldr x2, [x2, #3952] // x2 = callback (libnvjpeg+0x3f110, via R_AARCH64_RELATIVE at 0x59f70)
3c15c: f9447800 ldr x0, [x0, #2288] // x0 = mgr->block
3c160: f9401403 ldr x3, [x0, #40]
3c164: d63f0060 blr x3 // block->SetEventCallback(block, &mgr, callback)
The callback at libnvjpeg+0x3f110:
3f110: 7100043f cmp w1, #1 // only event type 1 does anything
3f114: 54000060 b.eq 0x3f120
3f118: 52800000 mov w0, #0
3f11c: d65f03c0 ret
3f120: a9bf7bf3 stp x19, x30, [sp, #-16]!
3f124: aa0003f3 mov x19, x0 // x0 = client context, used as if it were mgr
3f128: f9448c00 ldr x0, [x0, #2328]
3f12c: 97ff1715 bl NvOsSemaphoreSignal
3f130: f9449260 ldr x0, [x19, #2336]
3f134: 97ff1713 bl NvOsSemaphoreSignal
The callback is invoked synchronously from jpeg_write_raw_data → jpegTegraEncoderCompress (+0x3f7c8, block->vtbl[1](block, 1, ...)) → libnvmmlite → libnvmmlite_utils → libnvmmlite_image → callback.
In libnvmmlite_image (+0x4604..+0x46a4), command 1 runs the operation under a mutex. If the returned status is neither 0 nor 14, it calls the client callback with event 1 and a 24-byte payload {size=24, version=1, ..., status}:
4670: 7100027f cmp w19, #0
4674: 7a4e1a64 ccmp w19, #14, #4, ne // status == 0 or status == 14 -> return normally
4678: 54fffd80 b.eq 0x4628
467c: f9401ae0 ldr x0, [x23, #48] // client context (the dangling &mgr)
4680: d2800301 mov x1, #24
4684: f94036e4 ldr x4, [x23, #104] // client callback (libnvjpeg+0x3f110)
4688: f2c00021 movk x1, #1, lsl #32
468c: f9001be1 str x1, [sp, #48] // payload {size=24, version=1}
4694: b9003ff3 str w19, [sp, #60] // payload status
4698: 2a1603e1 mov w1, w22 // event = 1
46a0: d63f0080 blr x4
Evidence from the field crash (crashpad minidump)
Crash reason: SIGSEGV / SEGV_MAPERR
Crash address: 0x000000000000a99f
Thread "tile encoder" (crashed)
0 libpthread.so.0 + 0x9c00 pthread_mutex_lock, x0 = 0xa98f
1 libnvos.so + 0xa8c0 NvOsSemaphoreSignal(0xa98f)
2 libnvjpeg.so + 0x3f130 event callback (return address after the first signal)
...
NvJPEGEncoder::encodeFromFd (libnvmm_common, built from samples/common/classes/NvJpegEncoder.cpp)
- The callback’s context was
0xfffd65ff0ae8, an address on the crashing thread’s own stack. ctx+2328held0xa98fandctx+2336held another stack address, so neither is a semaphore.- The event payload was still on the stack: status = 4 in both dumps we have (two different units).
- All other threads were idle in condition-variable waits. Nothing else was inside libnvjpeg, NVENC or NvBufSurfTransform at the time.
- Other occurrences faulted at
0x9f0and0x9c8d, which are also garbage “semaphore” pointers.
Confirmed with gdb on a bench unit (same L4T)
- A breakpoint on the registration call shows
x1 = sp+88, a stack address.*(x1)is the heap-allocated manager. - A breakpoint on the callback’s event-1 path never fired in our bench runs (hundreds of normal encodes). So the bad pointer is harmless until a hardware operation fails, and we cannot yet make it fail on demand. In the field it failed after about 50 minutes, with the full camera, NVENC, VIC and CUDA pipeline running, on several units within about a minute of each other.
- A gdb breakpoint on the
block->vtbl[1](block, 1, ...)call injpegTegraEncoderCompress(+0x3f7c8) wedged the first encode. That point seems to be timing-sensitive.
R32.6.1’s libnvjpeg contains no calls to NvOsSemaphoreSignal at all, so this appears to be a new R35 code path.
Questions
- Can you confirm the
jpegTegraEncoderMgrCreatecallback registration bug, and is a fix planned for R35? The client data should presumably bemgr, not&mgron the stack. - What does operation status 4 mean for the JPEG encode block, and what can cause it on a 4096×2160 YUV420 pitch-linear NvBufSurface that encodes fine thousands of times beforehand? Knowing that would let us avoid the condition until there is a fix.