1000s of dmabuf open file handles reported by lsof

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

We have a process that does ~6 or so GStreamer encoder/decoder pipelines… Recently I mysteriously busted the system’s max open file limit. When I took a look at the process using lsof I noticed about 6000 (!) lines such as:

myproc 4903 user 2663u  a_inode               0,11         0     10329 dmabuf

I also noted ~250 for /dev/nvhost-vic, ~300 for /dev/nvhost-nvdec and another ~300 for /dev/nvhost-nvdec1

Is this normal/expected?

The process’s current max-open-files limit is at ~8000. Any issue (wrt these dmabuf handles) with increasing this limit?

2

Hi,
Please share your L4T release version and steps to reproduce the issue.

3

We’re on:

R32 (release), REVISION: 2.1, GCID: 16294929, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug 13 04:45:36 UTC 2019

We will do the work to come up with a minimal repro. Stay tuned and thank you.

4

Hi,
This may not be covered in our SQA tests, so it is probably an issue. We need to reproduce it first so that core teams can evaluate/check if we can include the usecase in future releases.

5

This issue reproduces in L4T 32.4.3 as well.

Here’s the test command line we used. The nvinfer configuration is derived from the sample test application of DeepSteeam.

$ gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_1080p_h264.mp4 ! \
        decodebin ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 \
        height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt ! \
        nvvideoconvert ! nvdsosd ! perf ! nvegltransform ! nveglglessink sync=false

Then, we aggregated the result of lsof while gst-launch is running as above.

About 7000 dmabuf handles are generated on the whole of the system.

$ lsof | grep dmabuf | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/120/gvfs
  Output information may be incomplete.
7075

Also the following is the breakdown of the result. The head number of each line shows the total occurrences of dmabuf handles for each command name lsof identifies. Dmabuf handles of compiz, gmain, gdbus and dconf\x20 reside regardless of the gst-launch execution. Others occur while gst-launch is running.

$ lsof | grep dmabuf | uniq -c --check-chars=9
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/120/gvfs
  Output information may be incomplete.
110 compiz     6110                 nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
110 gmain      6110  6121           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
110 gdbus      6110  6128           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
110 dconf\x20  6110  6130           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
630 gst-launc 11580                 nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 cuda-EvtH 11580 11582           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
1575 gst-launc 11580 11583           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 typefind: 11580 11600           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 gmain     11580 11601           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 qtdemux0: 11580 11602           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
630 multiqueu 11580 11603           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecBuf 11580 11605           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecDis 11580 11606           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecFrm 11580 11607           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecVPR 11580 11608           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 V4L2_DecT 11580 11609           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 gst-launc 11580 11610           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 m:src     11580 11611           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 nvv4l2dec 11580 11612           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
7

The discussion continues in