JetPack 7.2.1 (r39.2.1): NvRmFenceWait opens and closes a host1x device on every call

Platform: Jetson AGX Orin 64GB devkit, stock JetPack 7.2.1 from the apt feed
# R39 (release), REVISION: 2.1, GCID: 46758480, kernel 6.8.12-1021-tegra,
nvidia-l4t-core / nvidia-l4t-multimedia 39.2.1-20260806224157.

What happens

Every NvBufSurfTransform() on VIC, and every frame through the V4L2 encoder
(/dev/v4l2-nvenc), waits for its job with NvRmFenceWait(NULL, fence, timeout).
With a NULL device, libnvrm_host1x.so rebuilds everything it needs to wait, on every call:

openat("/sys/bus/platform/drivers/tegra-host1x", O_DIRECTORY)            = 24
getdents64(24, ...)                                                       = 208
openat(".../13e00000.host1x/numa_node")                                   = -1 ENOENT
openat(".../13e00000.host1x/13e00000.host1x.drm/drm", O_DIRECTORY)        = 25
getdents64(25, ...)                                                       = 112
openat("/dev/dri/renderD128", O_RDWR|O_CLOEXEC)                           = 26
getdents64(24, ...)                                                       = 0
openat("/sys/bus/platform/drivers/tegra-host1x", O_DIRECTORY)             = 24   (second walk)
... DRM_IOCTL_VERSION, wait ..., close(26)

With gdb on the exported symbols, each NvRmFenceWait(x0=NULL, x1={syncpt id, threshold}, x2=0xFFFFFFFF)
runs, in order: NvRmHost1xOpen → NvRmHost1xWaiterAllocate → NvRmHost1xSyncpointWait(new waiter, ...)
→ NvRmHost1xWaiterFree → NvRmHost1xClose. libnvbufsurftransform.so gets the function via dlsym;
the encoder libraries call it through their PLT from V4L2_EncThread, once per encoded frame.

Separately, NvRmHost1xStreamPushIncr / PushWords / PushBufferAddress each call getpid(),
about 33 times per conversion. glibc has not cached getpid since 2.25, so each is a syscall.

Reproducer

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <time.h>

#include "nvbufsurface.h"
#include "nvbufsurftransform.h"

static NvBufSurface *allocate(NvBufSurfaceColorFormat format) {
  NvBufSurfaceAllocateParams request = {0};
  request.params.width = 1920;
  request.params.height = 1536;
  request.params.colorFormat = format;
  request.params.layout = NVBUF_LAYOUT_PITCH;
  request.params.memType = NVBUF_MEM_SURFACE_ARRAY;
  NvBufSurface *surface = NULL;
  if (NvBufSurfaceAllocate(&surface, 1, &request) != 0 || surface == NULL) return NULL;
  surface->numFilled = 1;
  return surface;
}

static int64_t now_ns(void) {
  struct timespec t;
  clock_gettime(CLOCK_MONOTONIC, &t);
  return (int64_t)t.tv_sec * 1000000000 + t.tv_nsec;
}

int main(int argc, char **argv) {
  int calls = argc > 1 ? atoi(argv[1]) : 300;
  int per_second = argc > 2 ? atoi(argv[2]) : 30;
  NvBufSurfTransformConfigParams session = {0};
  session.compute_mode = NvBufSurfTransformCompute_VIC;
  if (NvBufSurfTransformSetSessionParams(&session) != NvBufSurfTransformError_Success) return 1;
  NvBufSurface *source = allocate(NVBUF_COLOR_FORMAT_YUYV);
  NvBufSurface *target = allocate(NVBUF_COLOR_FORMAT_NV12);
  if (source == NULL || target == NULL) return 1;
  NvBufSurfTransformParams conversion = {0};
  conversion.transform_flag = NVBUFSURF_TRANSFORM_FILTER;
  conversion.transform_filter = NvBufSurfTransformInter_Algo3;
  int64_t began = now_ns();
  int64_t in_calls = 0;
  int failed = 0;
  for (int i = 0; i < calls; ++i) {
    int64_t start = now_ns();
    if (NvBufSurfTransform(source, target, &conversion) != NvBufSurfTransformError_Success) ++failed;
    in_calls += now_ns() - start;
    if (per_second > 0) {
      int64_t due = began + (int64_t)(i + 1) * 1000000000 / per_second;
      struct timespec until = {due / 1000000000, due % 1000000000};
      clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &until, NULL);
    }
  }
  struct rusage used;
  getrusage(RUSAGE_SELF, &used);
  double user_ms = used.ru_utime.tv_sec * 1e3 + used.ru_utime.tv_usec / 1e3;
  double system_ms = used.ru_stime.tv_sec * 1e3 + used.ru_stime.tv_usec / 1e3;
  printf("%d NvBufSurfTransform calls, %d failed, %.2f ms wall per call, cpu per call %.3f ms (user %.3f, system %.3f)\n",
         calls, failed, (double)in_calls / calls / 1e6, (user_ms + system_ms) / calls, user_ms / calls,
         system_ms / calls);
  return failed != 0;
}
L=/usr/lib/aarch64-linux-gnu/nvidia
gcc -O2 -o repro repro.c -I/usr/src/jetson_multimedia_api/include -L$L \
    -lnvbufsurface -lnvbufsurftransform -Wl,-rpath,$L
./repro 900 30
strace -f -c ./repro 300 30
strace -f -e trace=openat,getdents64 ./repro 40 0

What it costs

Syscalls per NvBufSurfTransform (strace -f -c, 300 calls):

total ioctl getpid openat close getdents64
stock 92.4 30.8 33.1 7.3 7.0 5.3
with workaround 40.3 28.8 0 1.4 3.1 0.3

perf stat per call, stock devkit, 3 interleaved rounds of 900 calls at 30/s:

cycles instructions task-clock
stock 753–761 k 607–643 k 0.35–0.42 ms
with workaround 635–640 k 476 k 0.29 ms

In our real application (6× 1920×1536 YUYV cameras at 30 fps → VIC → NV12 → H.265 via /dev/v4l2-nvenc,
one process) the same workaround takes the process from 8.55 G to 7.14 G cycles per 20 s (−16.5 %) and
from 323 to 150 syscalls per frame, with the same frames and a bitstream that decodes cleanly either way.
In its profile, openat+getdents64+close were 8.9 % of the process’s cycles and getpid 4 %.

Workaround (to show the cost, not a proposed fix)

An LD_PRELOAD library that interposes NvRmFenceWait (and dlsym, only to hand the same function to
libnvbufsurftransform.so) and keeps one host1x handle and one waiter per thread, created with
NVIDIA’s own NvRmHost1xOpen / NvRmHost1xWaiterAllocate, then calls NvRmHost1xSyncpointWait on it;
plus a cached getpid. Source attached (nvshim.c). It relies on private signatures read from register
state, so it is only a measurement.

gcc -O2 -shared -fPIC -o nvshim.so nvshim.c -ldl -lpthread
LD_PRELOAD=$PWD/nvshim.so ./repro 900 30

Questions

  1. Is the per-call open/scan/close in NvRmFenceWait(NULL, …) intended on r39? Could it reuse a
    process- or thread-wide host1x handle and waiter, as the same thread already does for its other
    NvRmHost1xSyncpointWait calls?
  2. Can the getpid() calls in the NvRmHost1xStreamPush* path be removed or cached?
  3. Is there a supported way for an application to give NvBufSurfTransform / the V4L2 encoder a
    persistent host1x device so they do not take this path?

Related and smaller: per job, both buffers are imported and mapped into the channel
(DRM_IOCTL_PRIME_FD_TO_HANDLE, DRM_IOCTL_TEGRA_CHANNEL_MAP) and released afterwards even though they
are the same few buffers every frame (~5 % of our process). Is there a way to keep them pinned?

Hi,
This is expected due to hardware design. The workaround may trigger potential issue. Would suggest use existing NvBufSurface APIs. If you would like to do conversion asynchronously, please call NvBufSurfTransformAsync()