Platform: Jetson AGX Orin 64GB devkit, stock JetPack 7.2.1 from the apt feed
# R39 (release), REVISION: 2.1, GCID: 46758480, kernel 6.8.12-1021-tegra,
nvidia-l4t-core / nvidia-l4t-multimedia 39.2.1-20260806224157.
What happens
Every NvBufSurfTransform() on VIC, and every frame through the V4L2 encoder
(/dev/v4l2-nvenc), waits for its job with NvRmFenceWait(NULL, fence, timeout).
With a NULL device, libnvrm_host1x.so rebuilds everything it needs to wait, on every call:
openat("/sys/bus/platform/drivers/tegra-host1x", O_DIRECTORY) = 24
getdents64(24, ...) = 208
openat(".../13e00000.host1x/numa_node") = -1 ENOENT
openat(".../13e00000.host1x/13e00000.host1x.drm/drm", O_DIRECTORY) = 25
getdents64(25, ...) = 112
openat("/dev/dri/renderD128", O_RDWR|O_CLOEXEC) = 26
getdents64(24, ...) = 0
openat("/sys/bus/platform/drivers/tegra-host1x", O_DIRECTORY) = 24 (second walk)
... DRM_IOCTL_VERSION, wait ..., close(26)
With gdb on the exported symbols, each NvRmFenceWait(x0=NULL, x1={syncpt id, threshold}, x2=0xFFFFFFFF)
runs, in order: NvRmHost1xOpen → NvRmHost1xWaiterAllocate → NvRmHost1xSyncpointWait(new waiter, ...)
→ NvRmHost1xWaiterFree → NvRmHost1xClose. libnvbufsurftransform.so gets the function via dlsym;
the encoder libraries call it through their PLT from V4L2_EncThread, once per encoded frame.
Separately, NvRmHost1xStreamPushIncr / PushWords / PushBufferAddress each call getpid(),
about 33 times per conversion. glibc has not cached getpid since 2.25, so each is a syscall.
Reproducer
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <time.h>
#include "nvbufsurface.h"
#include "nvbufsurftransform.h"
static NvBufSurface *allocate(NvBufSurfaceColorFormat format) {
NvBufSurfaceAllocateParams request = {0};
request.params.width = 1920;
request.params.height = 1536;
request.params.colorFormat = format;
request.params.layout = NVBUF_LAYOUT_PITCH;
request.params.memType = NVBUF_MEM_SURFACE_ARRAY;
NvBufSurface *surface = NULL;
if (NvBufSurfaceAllocate(&surface, 1, &request) != 0 || surface == NULL) return NULL;
surface->numFilled = 1;
return surface;
}
static int64_t now_ns(void) {
struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
return (int64_t)t.tv_sec * 1000000000 + t.tv_nsec;
}
int main(int argc, char **argv) {
int calls = argc > 1 ? atoi(argv[1]) : 300;
int per_second = argc > 2 ? atoi(argv[2]) : 30;
NvBufSurfTransformConfigParams session = {0};
session.compute_mode = NvBufSurfTransformCompute_VIC;
if (NvBufSurfTransformSetSessionParams(&session) != NvBufSurfTransformError_Success) return 1;
NvBufSurface *source = allocate(NVBUF_COLOR_FORMAT_YUYV);
NvBufSurface *target = allocate(NVBUF_COLOR_FORMAT_NV12);
if (source == NULL || target == NULL) return 1;
NvBufSurfTransformParams conversion = {0};
conversion.transform_flag = NVBUFSURF_TRANSFORM_FILTER;
conversion.transform_filter = NvBufSurfTransformInter_Algo3;
int64_t began = now_ns();
int64_t in_calls = 0;
int failed = 0;
for (int i = 0; i < calls; ++i) {
int64_t start = now_ns();
if (NvBufSurfTransform(source, target, &conversion) != NvBufSurfTransformError_Success) ++failed;
in_calls += now_ns() - start;
if (per_second > 0) {
int64_t due = began + (int64_t)(i + 1) * 1000000000 / per_second;
struct timespec until = {due / 1000000000, due % 1000000000};
clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &until, NULL);
}
}
struct rusage used;
getrusage(RUSAGE_SELF, &used);
double user_ms = used.ru_utime.tv_sec * 1e3 + used.ru_utime.tv_usec / 1e3;
double system_ms = used.ru_stime.tv_sec * 1e3 + used.ru_stime.tv_usec / 1e3;
printf("%d NvBufSurfTransform calls, %d failed, %.2f ms wall per call, cpu per call %.3f ms (user %.3f, system %.3f)\n",
calls, failed, (double)in_calls / calls / 1e6, (user_ms + system_ms) / calls, user_ms / calls,
system_ms / calls);
return failed != 0;
}
L=/usr/lib/aarch64-linux-gnu/nvidia
gcc -O2 -o repro repro.c -I/usr/src/jetson_multimedia_api/include -L$L \
-lnvbufsurface -lnvbufsurftransform -Wl,-rpath,$L
./repro 900 30
strace -f -c ./repro 300 30
strace -f -e trace=openat,getdents64 ./repro 40 0
What it costs
Syscalls per NvBufSurfTransform (strace -f -c, 300 calls):
| total | ioctl | getpid | openat | close | getdents64 | |
|---|---|---|---|---|---|---|
| stock | 92.4 | 30.8 | 33.1 | 7.3 | 7.0 | 5.3 |
| with workaround | 40.3 | 28.8 | 0 | 1.4 | 3.1 | 0.3 |
perf stat per call, stock devkit, 3 interleaved rounds of 900 calls at 30/s:
| cycles | instructions | task-clock | |
|---|---|---|---|
| stock | 753–761 k | 607–643 k | 0.35–0.42 ms |
| with workaround | 635–640 k | 476 k | 0.29 ms |
In our real application (6× 1920×1536 YUYV cameras at 30 fps → VIC → NV12 → H.265 via /dev/v4l2-nvenc,
one process) the same workaround takes the process from 8.55 G to 7.14 G cycles per 20 s (−16.5 %) and
from 323 to 150 syscalls per frame, with the same frames and a bitstream that decodes cleanly either way.
In its profile, openat+getdents64+close were 8.9 % of the process’s cycles and getpid 4 %.
Workaround (to show the cost, not a proposed fix)
An LD_PRELOAD library that interposes NvRmFenceWait (and dlsym, only to hand the same function to
libnvbufsurftransform.so) and keeps one host1x handle and one waiter per thread, created with
NVIDIA’s own NvRmHost1xOpen / NvRmHost1xWaiterAllocate, then calls NvRmHost1xSyncpointWait on it;
plus a cached getpid. Source attached (nvshim.c). It relies on private signatures read from register
state, so it is only a measurement.
gcc -O2 -shared -fPIC -o nvshim.so nvshim.c -ldl -lpthread
LD_PRELOAD=$PWD/nvshim.so ./repro 900 30
Questions
- Is the per-call open/scan/close in NvRmFenceWait(NULL, …) intended on r39? Could it reuse a
process- or thread-wide host1x handle and waiter, as the same thread already does for its other
NvRmHost1xSyncpointWait calls? - Can the
getpid()calls in theNvRmHost1xStreamPush*path be removed or cached? - Is there a supported way for an application to give
NvBufSurfTransform/ the V4L2 encoder a
persistent host1x device so they do not take this path?
Related and smaller: per job, both buffers are imported and mapped into the channel
(DRM_IOCTL_PRIME_FD_TO_HANDLE, DRM_IOCTL_TEGRA_CHANNEL_MAP) and released afterwards even though they
are the same few buffers every frame (~5 % of our process). Is there a way to keep them pinned?