Hello everyone.
In my project, a number of processes runs in ROS pipeline on Xavier 32 Gb platform inside a docker container, and one of these processes uses VPI to process camera data. After a change in the pipeline (that effectively was merging the processing done in other non-VPI process into VPI process), I noticed a massive raise of minor page fault count in the pipeline (up to nearly 22-23k MPF per second). The investigation has shown that all the faults are evenly generated by four threads inside the VPI process. All these threads are named “Work queue”, and the only binary referencing this string was libnvvpi.so.
I have done perf trace for one of the threads, and here is the result.
perf trace output
1.049 ( 0.093 ms): mmap(arg0: 0, arg1: 1179648, arg2: 3, arg3: 1, arg4: 2046, arg5: 2228224) = 0x7d06ffe000
1.149 ( 0.046 ms): futex(arg0: 547351555576, arg1: 128, arg2: 2, arg3: 0, arg4: 1080, arg5: 2228224) = 0
1.201 ( 0.029 ms): futex(arg0: 547351555576, arg1: 128, arg2: 2, arg3: 0, arg4: 1080, arg5: 2228224) = 0
1.240 ( 0.004 ms): futex(arg0: 547351555576, arg1: 129, arg2: 1, arg3: 0, arg4: 547351555592, arg5: 383) = 0
1.345 ( 0.015 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: 3320604098773168717, arg5: 537650265928) = 0
1.378 ( 0.007 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 6188250391092097318, arg5: 547349744520) = 0
1.439 ( 0.006 ms): lseek(arg0: 4132, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 6188250391092097318, arg5: 547349744520) = 1179648
1.452 ( 0.006 ms): close(arg0: 4073, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
1.476 ( 0.007 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4131, arg3: -7781210708696532070, arg4: -6110083594271091604, arg5: 541678866777) = 0
1.493 ( 0.116 ms): munmap(arg0: 538752655360, arg1: 786432, arg2: -4096, arg3: 786432, arg4: 547594122664, arg5: 383) = 0
1.617 ( 0.027 ms): munmap(arg0: 537832075264, arg1: 393216, arg2: -4096, arg3: 393216, arg4: 547594122664, arg5: 383) = 0
1.650 ( 0.007 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1685, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1.662 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2176, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1.679 ( 0.022 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
2.385 ( 0.014 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
2.439 ( 0.008 ms): futex(arg0: 547351555576, arg1: 129, arg2: 1, arg3: 0, arg4: 547351555592, arg5: 383) = 1
3.726 ( 0.081 ms): mmap(arg0: 0, arg1: 786432, arg2: 3, arg3: 1, arg4: 1685, arg5: 0 ) = 0x7d392da000
3.813 ( 0.035 ms): mmap(arg0: 0, arg1: 393216, arg2: 3, arg3: 1, arg4: 2176, arg5: 786432) = 0x7d70292000
3.896 ( 0.010 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267472, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
3.910 ( 0.005 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267520, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
3.958 ( 0.010 ms): ioctl(arg0: 6, arg1: 1075334668, arg2: 537650266296, arg3: 43, arg4: 4294967295, arg5: 0) = 0
3.973 ( 0.083 ms): ioctl(arg0: 97, arg1: -1065334758, arg2: 537650265872, arg3: 0, arg4: 537650266756, arg5: 2035) = 0
4.067 ( 1.240 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650268640, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
5.342 ( 0.008 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267104, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
6.544 ( 0.037 ms): ioctl(arg0: 274, arg1: 3222292486, arg2: 537650263440, arg3: 1000000, arg4: 8300523052597248, arg5: 0) = 0
6.609 ( 0.014 ms): futex(arg0: 541641891260, arg1: 129, arg2: 2147483647, arg3: 0, arg4: 541641891184, arg5: 541641891232) = 1
6.631 (193.082 ms): futex(arg0: 541641891308, arg1: 128, arg2: 0, arg3: 0, arg4: 0, arg5: 541641891304) = 0
199.745 ( 0.009 ms): futex(arg0: 541641891168, arg1: 129, arg2: 1, arg3: 0, arg4: 541641891184, arg5: 383) = 0
199.785 ( 0.009 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -4962129251142984268, arg5: 537650265928) = 0
199.805 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 3320604098773168717, arg5: 547349744520) = 0
199.816 ( 0.004 ms): lseek(arg0: 4202, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 3320604098773168717, arg5: 547349744520) = 3407872
199.827 ( 0.004 ms): close(arg0: 4102, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
199.856 ( 0.006 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4196, arg3: -8670242527469985065, arg4: -7972866270244225058, arg5: 541678873177) = 0
199.872 ( 0.124 ms): munmap(arg0: 536989523968, arg1: 2228224, arg2: -4096, arg3: 2228224, arg4: 547594122664, arg5: 383) = 0
200.004 ( 0.053 ms): munmap(arg0: 536988344320, arg1: 1179648, arg2: -4096, arg3: 1179648, arg4: 547594122664, arg5: 383) = 0
200.064 ( 0.006 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1582, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
200.074 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2046, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
200.091 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
200.101 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
200.116 ( 0.192 ms): mmap(arg0: 0, arg1: 2228224, arg2: 3, arg3: 1, arg4: 1582, arg5: 0 ) = 0x7d0711e000
200.314 ( 0.218 ms): mmap(arg0: 0, arg1: 1179648, arg2: 3, arg3: 1, arg4: 2046, arg5: 2228224) = 0x7d06ffe000
200.591 ( 0.009 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: 2159768602618045944, arg5: 537650265928) = 0
200.612 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: -7972866270244225058, arg5: 547349744520) = 0
200.623 ( 0.004 ms): lseek(arg0: 4196, arg1: 0, arg2: 2, arg3: 537650265224, arg4: -7972866270244225058, arg5: 547349744520) = 1179648
200.634 ( 0.004 ms): close(arg0: 4102, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
200.652 ( 0.007 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4132, arg3: 1159261925789617192, arg4: 6188250391092097318, arg5: 541678866777) = 0
200.666 ( 0.052 ms): munmap(arg0: 537830203392, arg1: 786432, arg2: -4096, arg3: 786432, arg4: 547594122664, arg5: 383) = 0
200.724 ( 0.048 ms): munmap(arg0: 538752655360, arg1: 393216, arg2: -4096, arg3: 393216, arg4: 547594122664, arg5: 383) = 0
200.776 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1685, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
200.785 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2176, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
200.798 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
200.808 ( 0.004 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
200.820 ( 0.062 ms): mmap(arg0: 0, arg1: 786432, arg2: 3, arg3: 1, arg4: 1685, arg5: 0 ) = 0x7d392da000
200.890 ( 0.033 ms): mmap(arg0: 0, arg1: 393216, arg2: 3, arg3: 1, arg4: 2176, arg5: 786432) = 0x7d70292000
200.958 ( 0.010 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267472, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
200.972 ( 0.004 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267520, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
201.019 ( 0.010 ms): ioctl(arg0: 6, arg1: 1075334668, arg2: 537650266296, arg3: 43, arg4: 4294967295, arg5: 0) = 0
201.034 ( 0.076 ms): ioctl(arg0: 97, arg1: -1065334758, arg2: 537650265872, arg3: 0, arg4: 537650266756, arg5: 2036) = 0
201.122 ( 1.172 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650268640, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
202.321 ( 0.010 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267104, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
202.379 ( 0.006 ms): ioctl(arg0: 274, arg1: 3222292486, arg2: 537650263440, arg3: 1000000, arg4: 8300523052597248, arg5: 0) = 0
202.405 ( 0.012 ms): futex(arg0: 541641891256, arg1: 129, arg2: 2147483647, arg3: 0, arg4: 541641891184, arg5: 541641891232) = 1
202.425 (198.383 ms): futex(arg0: 541641891304, arg1: 128, arg2: 0, arg3: 0, arg4: 0, arg5: 541641891304) = 0
400.834 ( 0.010 ms): futex(arg0: 541641891168, arg1: 129, arg2: 1, arg3: 0, arg4: 541641891184, arg5: 383) = 0
400.883 ( 0.009 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: 8535673376665544345, arg5: 537650265928) = 0
400.908 ( 0.007 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 2159768602618045944, arg5: 547349744520) = 0
400.939 ( 0.005 ms): lseek(arg0: 4229, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 2159768602618045944, arg5: 547349744520) = 3407872
400.953 ( 0.005 ms): close(arg0: 4036, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
400.980 ( 0.006 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4202, arg3: -3919632870910507970, arg4: 3320604098773168717, arg5: 541678873177) = 0
400.996 ( 0.136 ms): munmap(arg0: 536989523968, arg1: 2228224, arg2: -4096, arg3: 2228224, arg4: 547594122664, arg5: 383) = 0
401.140 ( 0.081 ms): munmap(arg0: 536988344320, arg1: 1179648, arg2: -4096, arg3: 1179648, arg4: 547594122664, arg5: 383) = 0
401.227 ( 0.006 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1582, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
401.237 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2046, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
401.252 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
401.294 ( 0.020 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
401.325 ( 0.235 ms): mmap(arg0: 0, arg1: 2228224, arg2: 3, arg3: 1, arg4: 1582, arg5: 0 ) = 0x7d0711e000
401.568 ( 0.094 ms): mmap(arg0: 0, arg1: 1179648, arg2: 3, arg3: 1, arg4: 2046, arg5: 2228224) = 0x7d06ffe000
401.718 ( 0.009 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -4962129251142984268, arg5: 537650265928) = 0
401.738 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 3320604098773168717, arg5: 547349744520) = 0
401.748 ( 0.004 ms): lseek(arg0: 4202, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 3320604098773168717, arg5: 547349744520) = 1179648
401.758 ( 0.004 ms): close(arg0: 4036, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
401.778 ( 0.006 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4196, arg3: -8670242527469985065, arg4: -7972866270244225058, arg5: 541678866777) = 0
401.792 ( 0.051 ms): munmap(arg0: 537830203392, arg1: 786432, arg2: -4096, arg3: 786432, arg4: 547594122664, arg5: 383) = 0
401.849 ( 0.031 ms): munmap(arg0: 538752655360, arg1: 393216, arg2: -4096, arg3: 393216, arg4: 547594122664, arg5: 383) = 0
401.884 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1685, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
401.893 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2176, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
401.927 ( 0.007 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
401.938 ( 0.005 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
401.952 ( 0.091 ms): mmap(arg0: 0, arg1: 786432, arg2: 3, arg3: 1, arg4: 1685, arg5: 0 ) = 0x7d392da000
402.049 ( 0.037 ms): mmap(arg0: 0, arg1: 393216, arg2: 3, arg3: 1, arg4: 2176, arg5: 786432) = 0x7d70292000
402.144 ( 0.010 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267472, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
402.159 ( 0.006 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267520, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
402.213 ( 0.009 ms): ioctl(arg0: 6, arg1: 1075334668, arg2: 537650266296, arg3: 43, arg4: 4294967295, arg5: 0) = 0
402.227 ( 0.077 ms): ioctl(arg0: 97, arg1: -1065334758, arg2: 537650265872, arg3: 0, arg4: 537650266756, arg5: 2037) = 0
402.315 ( 1.141 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650268640, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
403.483 ( 0.033 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267104, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
403.566 ( 0.008 ms): ioctl(arg0: 274, arg1: 3222292486, arg2: 537650263440, arg3: 1000000, arg4: 8300523052597248, arg5: 0) = 0
403.593 ( 0.013 ms): futex(arg0: 541641891260, arg1: 129, arg2: 2147483647, arg3: 0, arg4: 541641891184, arg5: 541641891232) = 1
403.614 (197.426 ms): futex(arg0: 541641891308, arg1: 128, arg2: 0, arg3: 0, arg4: 0, arg5: 541641891304) = 0
601.063 ( 0.008 ms): futex(arg0: 541641891168, arg1: 129, arg2: 1, arg3: 0, arg4: 541641891184, arg5: 383) = 0
601.109 ( 0.010 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -7972866270244225058, arg5: 537650265928) = 0
601.134 ( 0.007 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: -4962129251142984268, arg5: 547349744520) = 0
601.146 ( 0.004 ms): lseek(arg0: 4201, arg1: 0, arg2: 2, arg3: 537650265224, arg4: -4962129251142984268, arg5: 547349744520) = 3407872
601.157 ( 0.004 ms): close(arg0: 4054, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
601.179 ( 0.006 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4229, arg3: -4440404945469654319, arg4: 2159768602618045944, arg5: 541678873177) = 0
601.194 ( 0.117 ms): munmap(arg0: 536989523968, arg1: 2228224, arg2: -4096, arg3: 2228224, arg4: 547594122664, arg5: 383) = 0
601.318 ( 0.055 ms): munmap(arg0: 536988344320, arg1: 1179648, arg2: -4096, arg3: 1179648, arg4: 547594122664, arg5: 383) = 0
601.378 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1582, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
601.387 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2046, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
601.401 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
601.411 ( 0.005 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
601.424 ( 0.181 ms): mmap(arg0: 0, arg1: 2228224, arg2: 3, arg3: 1, arg4: 1582, arg5: 0 ) = 0x7d0711e000
601.611 ( 0.101 ms): mmap(arg0: 0, arg1: 1179648, arg2: 3, arg3: 1, arg4: 2046, arg5: 2228224) = 0x7d06ffe000
601.755 ( 0.008 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: 2979044877427254504, arg5: 537650265928) = 0
601.771 ( 0.007 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 2159768602618045944, arg5: 547349744520) = 0
601.781 ( 0.004 ms): lseek(arg0: 4228, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 2159768602618045944, arg5: 547349744520) = 1179648
601.790 ( 0.004 ms): close(arg0: 4054, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
601.809 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4202, arg3: -3919632870910507970, arg4: 3320604098773168717, arg5: 541678866777) = 0
601.822 ( 0.063 ms): munmap(arg0: 537830203392, arg1: 786432, arg2: -4096, arg3: 786432, arg4: 547594122664, arg5: 383) = 0
601.890 ( 0.024 ms): munmap(arg0: 538752655360, arg1: 393216, arg2: -4096, arg3: 393216, arg4: 547594122664, arg5: 383) = 0
601.918 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1685, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
601.927 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2176, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
601.940 ( 0.005 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
601.949 ( 0.028 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
601.984 ( 0.064 ms): mmap(arg0: 0, arg1: 786432, arg2: 3, arg3: 1, arg4: 1685, arg5: 0 ) = 0x7d392da000
602.053 ( 0.056 ms): mmap(arg0: 0, arg1: 393216, arg2: 3, arg3: 1, arg4: 2176, arg5: 786432) = 0x7d70292000
602.141 ( 0.010 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267472, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
602.155 ( 0.004 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267520, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
602.201 ( 0.008 ms): ioctl(arg0: 6, arg1: 1075334668, arg2: 537650266296, arg3: 43, arg4: 4294967295, arg5: 0) = 0
602.215 ( 0.070 ms): ioctl(arg0: 97, arg1: -1065334758, arg2: 537650265872, arg3: 0, arg4: 537650266756, arg5: 2038) = 0
602.295 ( 1.062 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650268640, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
603.383 ( 0.009 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267104, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
603.441 ( 0.006 ms): ioctl(arg0: 274, arg1: 3222292486, arg2: 537650263440, arg3: 1000000, arg4: 8300523052597248, arg5: 0) = 0
603.466 ( 0.012 ms): futex(arg0: 541641891256, arg1: 129, arg2: 2147483647, arg3: 0, arg4: 541641891184, arg5: 541641891232) = 1
603.484 (197.072 ms): futex(arg0: 541641891304, arg1: 128, arg2: 0, arg3: 0, arg4: 0, arg5: 541641891304) = 0
800.830 ( 0.055 ms): futex(arg0: 541641891168, arg1: 129, arg2: 1, arg3: 0, arg4: 541641891184, arg5: 383) = 0
801.056 ( 0.049 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -8444614308519573307, arg5: 537650265928) = 0
801.242 ( 0.069 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 6188250391092097318, arg5: 547349744520) = 0
801.479 ( 0.066 ms): lseek(arg0: 2412, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 6188250391092097318, arg5: 547349744520) = 3407872
801.672 ( 0.043 ms): close(arg0: 4072, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
801.850 ( 0.041 ms): futex(arg0: 547351555576, arg1: 129, arg2: 1, arg3: 0, arg4: 547351555592, arg5: 383) = 1
802.067 ( 0.044 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4201, arg3: 44277986174384916, arg4: -4962129251142984268, arg5: 541678873177) = 0
802.240 ( 0.047 ms): futex(arg0: 547351555576, arg1: 129, arg2: 1, arg3: 0, arg4: 547351555592, arg5: 383) = 1
802.416 ( 0.305 ms): munmap(arg0: 536989523968, arg1: 2228224, arg2: -4096, arg3: 2228224, arg4: 547594122664, arg5: 383) = 0
802.855 ( 0.118 ms): munmap(arg0: 536988344320, arg1: 1179648, arg2: -4096, arg3: 1179648, arg4: 547594122664, arg5: 383) = 0
803.129 ( 0.044 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1582, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
803.404 ( 0.045 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2046, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
803.580 ( 0.019 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
803.606 ( 0.008 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
803.624 ( 0.175 ms): mmap(arg0: 0, arg1: 2228224, arg2: 3, arg3: 1, arg4: 1582, arg5: 0 ) = 0x7d0711e000
803.806 ( 0.084 ms): mmap(arg0: 0, arg1: 1179648, arg2: 3, arg3: 1, arg4: 2046, arg5: 2228224) = 0x7d06ffe000
803.955 ( 0.020 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -7972866270244225058, arg5: 537650265928) = 0
803.996 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 6919546698002187740, arg5: 547349744520) = 0
804.007 ( 0.004 ms): lseek(arg0: 4201, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 6919546698002187740, arg5: 547349744520) = 1179648
804.018 ( 0.004 ms): close(arg0: 4102, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
804.040 ( 0.007 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4228, arg3: -4440404945469654319, arg4: 2159768602618045944, arg5: 541678866777) = 0
804.056 ( 0.046 ms): munmap(arg0: 537830203392, arg1: 786432, arg2: -4096, arg3: 786432, arg4: 547594122664, arg5: 383) = 0
804.106 ( 0.024 ms): munmap(arg0: 538752655360, arg1: 393216, arg2: -4096, arg3: 393216, arg4: 547594122664, arg5: 383) = 0
804.135 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1685, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
804.144 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2176, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
804.157 ( 0.030 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
804.192 ( 0.005 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
804.204 ( 0.065 ms): mmap(arg0: 0, arg1: 786432, arg2: 3, arg3: 1, arg4: 1685, arg5: 0 ) = 0x7d392da000
804.275 ( 0.035 ms): mmap(arg0: 0, arg1: 393216, arg2: 3, arg3: 1, arg4: 2176, arg5: 786432) = 0x7d70292000
804.366 ( 0.010 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267472, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
804.380 ( 0.004 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267520, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
804.428 ( 0.013 ms): ioctl(arg0: 6, arg1: 1075334668, arg2: 537650266296, arg3: 43, arg4: 4294967295, arg5: 0) = 0
804.447 ( 0.085 ms): ioctl(arg0: 97, arg1: -1065334758, arg2: 537650265872, arg3: 0, arg4: 537650266756, arg5: 2039) = 0
804.543 ( 3.350 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650268640, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
807.936 ( 0.014 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267104, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
808.015 ( 0.007 ms): ioctl(arg0: 274, arg1: 3222292486, arg2: 537650263440, arg3: 1000000, arg4: 8300523052597248, arg5: 0) = 0
808.043 ( 0.014 ms): futex(arg0: 541641891260, arg1: 129, arg2: 2147483647, arg3: 0, arg4: 541641891184, arg5: 541641891232) = 1
808.065 (192.486 ms): futex(arg0: 541641891308, arg1: 128, arg2: 0, arg3: 0, arg4: 0, arg5: 541641891304) = 0
1000.815 ( 0.051 ms): futex(arg0: 541641891168, arg1: 129, arg2: 1, arg3: 0, arg4: 541641891184, arg5: 383) = 0
1001.047 ( 0.047 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -4962129251142984268, arg5: 537650265928) = 0
1001.265 ( 0.046 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: -7972866270244225058, arg5: 547349744520) = 0
1001.463 ( 0.062 ms): lseek(arg0: 4227, arg1: 0, arg2: 2, arg3: 537650265224, arg4: -7972866270244225058, arg5: 547349744520) = 3407872
1001.631 ( 0.043 ms): close(arg0: 4099, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
1001.870 ( 0.044 ms): futex(arg0: 547351555576, arg1: 129, arg2: 1, arg3: 0, arg4: 547351555592, arg5: 383) = 1
1002.059 ( 0.046 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2412, arg3: 1159261925789617192, arg4: 6188250391092097318, arg5: 541678873177) = 0
1002.238 ( 0.275 ms): munmap(arg0: 536989523968, arg1: 2228224, arg2: -4096, arg3: 2228224, arg4: 547594122664, arg5: 383) = 0
1002.934 ( 0.210 ms): munmap(arg0: 536988344320, arg1: 1179648, arg2: -4096, arg3: 1179648, arg4: 547594122664, arg5: 383) = 0
1004.179 ( 0.072 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1582, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1004.438 ( 0.055 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2046, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1004.622 ( 0.049 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
1004.989 ( 0.049 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
1005.251 ( 0.404 ms): mmap(arg0: 0, arg1: 2228224, arg2: 3, arg3: 1, arg4: 1582, arg5: 0 ) = 0x7d0711e000
1006.050 ( 0.151 ms): mmap(arg0: 0, arg1: 1179648, arg2: 3, arg3: 1, arg4: 2046, arg5: 2228224) = 0x7d06ffe000
1006.450 ( 0.050 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -8444614308519573307, arg5: 537650265928) = 0
1006.650 ( 0.046 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: 6188250391092097318, arg5: 547349744520) = 0
1006.835 ( 0.040 ms): lseek(arg0: 4099, arg1: 0, arg2: 2, arg3: 537650265224, arg4: 6188250391092097318, arg5: 547349744520) = 1179648
1007.084 ( 0.045 ms): close(arg0: 2412, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
1007.935 ( 0.067 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4201, arg3: -480610389536991543, arg4: 6919546698002187740, arg5: 541678866777) = 0
1008.190 ( 0.115 ms): munmap(arg0: 537830203392, arg1: 786432, arg2: -4096, arg3: 786432, arg4: 547594122664, arg5: 383) = 0
1008.504 ( 0.095 ms): munmap(arg0: 538752655360, arg1: 393216, arg2: -4096, arg3: 393216, arg4: 547594122664, arg5: 383) = 0
1008.742 ( 0.063 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1685, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1008.958 ( 0.042 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2176, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1009.110 ( 0.046 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
1009.751 ( 0.077 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
1010.020 ( 0.142 ms): mmap(arg0: 0, arg1: 786432, arg2: 3, arg3: 1, arg4: 1685, arg5: 0 ) = 0x7d392da000
1010.340 ( 0.137 ms): mmap(arg0: 0, arg1: 393216, arg2: 3, arg3: 1, arg4: 2176, arg5: 786432) = 0x7d70292000
1010.688 ( 0.060 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267472, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
1010.917 ( 0.046 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267520, arg3: 547593901608, arg4: 1080, arg5: 0) = 0
1011.102 ( 0.053 ms): ioctl(arg0: 6, arg1: 1075334668, arg2: 537650266296, arg3: 43, arg4: 4294967295, arg5: 0) = 0
1011.324 ( 0.102 ms): ioctl(arg0: 97, arg1: -1065334758, arg2: 537650265872, arg3: 0, arg4: 537650266756, arg5: 2040) = 0
1011.449 ( 2.410 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650268640, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
1013.899 ( 0.010 ms): ioctl(arg0: 4, arg1: -1072674810, arg2: 537650267104, arg3: 547593901608, arg4: 541762222192, arg5: 383) = 0
1013.971 ( 0.006 ms): ioctl(arg0: 274, arg1: 3222292486, arg2: 537650263440, arg3: 1000000, arg4: 8300523052597248, arg5: 0) = 0
1013.998 ( 0.011 ms): futex(arg0: 541641891256, arg1: 129, arg2: 2147483647, arg3: 0, arg4: 541641891184, arg5: 541641891232) = 1
1014.016 (187.978 ms): futex(arg0: 541641891304, arg1: 128, arg2: 0, arg3: 0, arg4: 0, arg5: 541641891304) = 0
1202.020 ( 0.008 ms): futex(arg0: 541641891168, arg1: 129, arg2: 1, arg3: 0, arg4: 541641891184, arg5: 383) = 0
1202.052 ( 0.007 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -5012239241000435416, arg5: 537650265928) = 0
1202.070 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: -8444614308519573307, arg5: 547349744520) = 0
1202.080 ( 0.004 ms): lseek(arg0: 4132, arg1: 0, arg2: 2, arg3: 537650265224, arg4: -8444614308519573307, arg5: 547349744520) = 3407872
1202.090 ( 0.004 ms): close(arg0: 4053, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
1202.110 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4227, arg3: -8670242527469985065, arg4: -7972866270244225058, arg5: 541678873177) = 0
1202.123 ( 0.098 ms): munmap(arg0: 536989523968, arg1: 2228224, arg2: -4096, arg3: 2228224, arg4: 547594122664, arg5: 383) = 0
1202.227 ( 0.059 ms): munmap(arg0: 536988344320, arg1: 1179648, arg2: -4096, arg3: 1179648, arg4: 547594122664, arg5: 383) = 0
1202.291 ( 0.005 ms): ioctl(arg0: 6, arg1: 19972, arg2: 1582, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1202.300 ( 0.004 ms): ioctl(arg0: 6, arg1: 19972, arg2: 2046, arg3: 547816296288, arg4: 547594122664, arg5: 383) = 0
1202.314 ( 0.005 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
1202.324 ( 0.005 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265528, arg3: 537650265528, arg4: 547349751712, arg5: 383) = 0
1202.336 ( 0.165 ms): mmap(arg0: 0, arg1: 2228224, arg2: 3, arg3: 1, arg4: 1582, arg5: 0 ) = 0x7d0711e000
1202.507 ( 0.089 ms): mmap(arg0: 0, arg1: 1179648, arg2: 3, arg3: 1, arg4: 2046, arg5: 2228224) = 0x7d06ffe000
1202.672 ( 0.011 ms): ioctl(arg0: 6, arg1: -1073197553, arg2: 537650265560, arg3: 547349571640, arg4: -4962129251142984268, arg5: 537650265928) = 0
1202.692 ( 0.006 ms): ioctl(arg0: 6, arg1: -1073197552, arg2: 537650265224, arg3: 537650265224, arg4: -7972866270244225058, arg5: 547349744520) = 0
1202.702 ( 0.004 ms): lseek(arg0: 4227, arg1: 0, arg2: 2, arg3: 537650265224, arg4: -7972866270244225058, arg5: 547349744520) = 1179648
1202.712 ( 0.004 ms): close(arg0: 4053, arg1: 0, arg2: 537650273928, arg3: 1, arg4: 547349751712, arg5: 383) = 0
1202.724 ( 0.042 ms): futex(arg0: 547351555576, arg1: 128, arg2: 2, arg3: 0, arg4: 537650266936, arg5: 383) = 0
1202.783 ( 0.012 ms): futex(arg0: 547351555576, arg1: 129, arg2: 1, arg3: 0, arg4: 547351555592, arg5: 383) = 1
1202.808 ( 0.007 ms): ioctl(arg0: 6, arg1: 19972, arg2: 4099, arg3: 1159261925789617192, arg4: 6188250391092097318, arg5: 541678866777) = 0
1202.823 ( 0.009 ms): futex(arg0: 547351555576, arg1: 129, arg2: 1, arg3: 0, arg4: 547351555592, arg5: 383) = 1
1202.837 ( 0.047 ms): munmap(arg0: 537830203392, arg1: 786432, arg2: -4096, arg3: 786432, arg4: 547594122664, arg5: 383) = 0
1203.058 ( 0.032 ms): munmap(arg0: 538752655360, arg1: 393216, arg2: -4096, arg3: 393216, arg4: 547594122664, arg5: 383) = 0
The trace shows the frame processing which is done every 200 ms. Among other syscalls, we see a number of mmap/mmunmap done. The total amount of mmap’ed memory per one frame is 2228224+1179648+786432+393216 = 4587520 bytes, or 1120 pages. For 5 frames per seconds and 4 threads, this is 22240 pages per second which matches the amount of PFs we see.
These allocations are not seen in our heap allocation profiles so we think mmaps are done not by the allocator but by the library itself. Moreover, we tried to allocate an arena for the allocator preliminary and it doesn’t help.
Is there a way we can avoid these page faults? Or does it indicate a problem in the library itself showing under some conditions?
Environment: Ubuntu 20.04 in nvidia-docker, Linux 4.9.253-l4t-32.7.1-51-g6aa06bd, Jetpack-based.
If any additional information is required, please tell me.
Thank you in advance.