Tegra System Profiler missing CUDA traces with TX1

I am seeing intermittent behavior with Tegra System Profiler generating traces/timelines for CUDA. Other timelines (including NVTX) are consistently created. However, I have regularly seen where CUDA traces will be generated as part of the report, and then suddenly it will stop generating the traces. I cannot find the reason why. I have verified the environment variables on the Tegra each time the problem has occurred and am certain that this is not the problem. In one case, the profiler stopped generating the CUDA trace in mid-run, creating a partial trace. In some cases, I have been able to restore creation of the traces with a Tegra reboot and a complete clean/build of the code. In other cases, this has not worked. Is anyone aware of a bug regarding the loss of these traces?

Hi TSkyMan,

Not sure whether that’s issue on old version of tool, or there could be other bugs at play.
Please help to provide your BSP, CUDA toolkit and Tegra System Profiler version that you used for our reference.
Then we could try to investigate the issue you met.

Thanks

I am also trying to get CUDA timelines from the Tegra System Profiler and seeing the following warnings:

Injection	nvidia@192.168.1.12 (:0:4)	00:00.000	
Common injection library initialized successfully.
Information	Injection	nvidia@192.168.1.12 (:0:4)	00:00.000	
OpenGL injection initialized successfully.
Information	Injection	nvidia@192.168.1.12 (:0:4)	00:00.000	
CUDA injection initialized successfully
Warning	Analysis		00:42.993	
No OpenGL events were collected. Does your application use OpenGL?
Information	Analysis		00:42.993	
Number of OpenGL events collected: 0.
Warning	Analysis		00:42.993	
No CUDA events were collected. Does your application use CUDA?
Information	Analysis		00:42.993	
Number of CUDA events collected: 0.
Information	Analysis		00:42.993	
Number of GPU context switch events collected: 65.
Information	Analysis		00:42.993	
Finishing profiling

The one I am specifically concerned about is, “No CUDA events were collected. Does your application use CUDA?”.

I am launching a CUDA based application on my TX2 system.

I am using Tegra System Profiler version 3.8.450-a527a18 Linux.

The CUDA version on my TX2 is V8.0.72

My BSP version:

cat /etc/nv_tegra_release 
# R28 (release), REVISION: 1.0, GCID: 9379712, BOARD: t186ref, EABI: aarch64, DATE: Thu Jul 20 07:59:31 UTC 2017
7f8fb47183cbd3d4cacc5eb50fc7869aacdefe40 */usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvideocodec.so
c33af0b5f9a88ab8ff62b31c16c635aa2744902d */usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvidconv.so
678e42e35687f11c9b5b602a539a56cc3de1188f */usr/lib/aarch64-linux-gnu/tegra/libnvomx.so
ccecdd04fb3ef95308a25a96ccf6670310400ba1 */usr/lib/aarch64-linux-gnu/tegra/libnveglstreamproducer.so
1653e5d266b7b030dc882d226b03f0c0157b4f3a */usr/lib/aarch64-linux-gnu/tegra/libnvtx_helper.so
fc646aa1d227d10ee3c338373e721ffade0b36d5 */usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so
c255aeebc742731b2a1e796816178b3b8ffa7dea */usr/lib/aarch64-linux-gnu/tegra/libglx.so
6cc4acdeee4908f9c4ff4f4dcfedef71be190354 */usr/lib/aarch64-linux-gnu/tegra/libargus_socketserver.so
5b66cf6e49430ca8918835a8ea287b51c8b3b941 */usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so
9f8b91a4b08d160d5f473b43e9874a5c24c66e88 */usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so
a3a2931e3fe2e5d40f6783d7d6ce3639ac88f0ed */usr/lib/aarch64-linux-gnu/tegra/libnvwinsys.so
7259ca958e6e595bfd3f50b914b9f51b12419ba8 */usr/lib/aarch64-linux-gnu/tegra/libargus.so
1a599b8a1f7e5abbbc8b3e51d3f48bcc5124f51a */usr/lib/aarch64-linux-gnu/tegra/libnvmm.so
97cf051cc8ac5aecf158bc3c85feb83b89fefdd9 */usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
3c48f3b81b1b7b333df5b261dc920736194e1f95 */usr/lib/aarch64-linux-gnu/tegra/libnvexif.so
cbe774108f73fe79b48ae7357e08e9413344dabc */usr/lib/aarch64-linux-gnu/tegra/libnvdc.so
8a9db15bf96f8c89967e96f1f55a116f0ad8853c */usr/lib/aarch64-linux-gnu/tegra/libnvavp.so
429e26afbfaf88b8ace2e0fa35207a2499cec9e1 */usr/lib/aarch64-linux-gnu/tegra/libnvtestresults.so
3f48c67f1c1650df51586078d1d8d3bd3740b025 */usr/lib/aarch64-linux-gnu/tegra/libargus_socketclient.so
4afd017ec9e9a16138da168c29a70f6bef4bd868 */usr/lib/aarch64-linux-gnu/tegra/libnvmm_utils.so
4336787797e9727d6fead71a027a5b5a10105a21 */usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so
0eeac3a25c46c2095db087f69fcb2da8e7c51855 */usr/lib/aarch64-linux-gnu/tegra/libnvll.so
1f2fc2642f5cd373b5db26921d85014b6abf840d */usr/lib/aarch64-linux-gnu/tegra/libnvcameratools.so
d5c04359d52a3d594fa091a18653426262a7197a */usr/lib/aarch64-linux-gnu/tegra/libnvapputil.so
8f47a0da6cc1e75b4ebd2ae33f83503249b6d5ee */usr/lib/aarch64-linux-gnu/tegra/libnveglstream_camconsumer.so
0667aa3962b8eda69023ac17830a2efc016cad4f */usr/lib/aarch64-linux-gnu/tegra/libnvrm.so
8653db228561f903a452d4cab1cc632ca3315881 */usr/lib/aarch64-linux-gnu/tegra/libnvcam_imageencoder.so
fd427c65f562573a5826fd29fc5410f0290ad52f */usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so
183da5b0281e0ee120545e2eaa99f56a0bb89d02 */usr/lib/aarch64-linux-gnu/tegra/libnvparser.so
beb1786a7d0e9464e98bdf3dda5d11c994069b8a */usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so
07c1e569a35cb39c77728ecbb7212f6339c8fd68 */usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so
ae214a66a4fe6ef66c15ea40a0a03dadb8055f72 */usr/lib/aarch64-linux-gnu/tegra/libnvtnr.so
ad4f99d3c3a6daa5829678a0defd1b2345b1c3b1 */usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so
cb725c103def5f9c8f0e25205b3b39eab4642721 */usr/lib/aarch64-linux-gnu/tegra/libnvidia-egl-wayland.so
9673606cfb805c3e1563fcdf1256cfb6c95fecc9 */usr/lib/aarch64-linux-gnu/tegra/libnvfnetstoredefog.so
f213ecec058176a1830e0621907f28176f57ff7e */usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so
a54283f9ed83ef15d6bd433d97e6a53e73176219 */usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so
532626aba510a1b8d586c04b23011ad7f48ff351 */usr/lib/aarch64-linux-gnu/tegra/libnvcolorutil.so
cc9f715b1fd1b9719ff845f1b9c07c3f3162fe2e */usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so
24237d67a163325fa3bf5194c5934bb944f71b58 */usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so
24a0ba50281234b4fe3205032a39eb88b89d2fd5 */usr/lib/aarch64-linux-gnu/tegra/libnvmm_contentpipe.so
cde93d596b8976787dc3f5f5bff1a3ec49dc8a13 */usr/lib/aarch64-linux-gnu/tegra/libnvimp.so
340c90216c7662a2ae6df0d9f5db961c9c7b3752 */usr/lib/aarch64-linux-gnu/tegra/libnvos.so
331481e2895586a29de020f40a1a288e2fc8d58b */usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so
56d4dd97a4073b605a4c906caaee0224affda2a4 */usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so
7c627627fbc26a280c5f395ba04ab01891f4341f */usr/lib/aarch64-linux-gnu/tegra/libnvfnetstorehdfx.so
0c3deb2a856368700fcc226110238e3299054b1b */usr/lib/aarch64-linux-gnu/tegra/libnvomxilclient.so
4f20b8cc95d69177ce108423cf5bac116e58a8c9 */usr/lib/aarch64-linux-gnu/tegra/libnvcamlog.so
1afa41bd35fc74e4f978875c0a6db0a8997201e5 */usr/lib/aarch64-linux-gnu/tegra/libnvmm_parser.so
d95121ac07e17d56500763b74d2fb29159fea85d */usr/lib/aarch64-linux-gnu/tegra/libscf.so
65acd5f0844c6dc12b71cf6fa46baf0d6c8e9a70 */usr/lib/aarch64-linux-gnu/tegra/libnvosd.so
3ef04ac64cac4cbe8f5c25414d2b71373d3a99a0 */usr/lib/xorg/modules/drivers/nvidia_drv.so
c255aeebc742731b2a1e796816178b3b8ffa7dea */usr/lib/xorg/modules/extensions/libglx.so

I have set the following environment variables in my .bashrc

export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:
export LD_PRELOAD=/opt/nvidia/tegra_system_profiler/libLauncher64.so
export CUDA_INJECTION64_PATH=/opt/nvidia/tegra_system_profiler/libToolsInjection64.so

Hi ceres.imaging,

Please find a separate topic into TX2 board - https://devtalk.nvidia.com/default/board/188/jetson-tx2/

Thanks