Hello Nsight Systems team,
Could you clarify the counter definitions and completeness-warning conditions in Nsight Systems 2026.5.1.161? A small CUDA probe has complete coverage of its 32 expected kernels, but reports 112 CUPTI events produced, 100 CUDA events collected, and a completeness warning. We have not established whether this reflects category/range filtering or actual loss of other records.
Environment and workload
- Nsight Systems CLI 2026.5.1.161-265138896106v0; diagnostic confirms the bundled libcupti.so.13.4 was loaded.
- Windows host, Docker Linux container, Ubuntu 24.04.3/CUDA 13.0.1, GeForce RTX 4060 Ti.
- Saved nvidia-smi output: NVIDIA-SMI 615.71.08, KMD 616.92, CUDA UMD 13.4. These are the labels printed by the tool, not interchangeable version identifiers.
- One target process and one nonblocking CUDA stream; no managed memory or CUDA graphs. 256 uint32 elements are zeroed and synchronized before capture.
- The application calls cudaProfilerStart(), launches 32 simple increment kernels, synchronizes the stream, then calls cudaProfilerStop(). Validation/copy and resource cleanup are after the capture range. All 256 output values are 32, with zero application CUDA-call/validation errors.
The capture command, with local paths replaced, is:
nsys profile --trace=cuda-sw --sample=none --cpuctxsw=none \
--capture-range=cudaProfilerApi --capture-range-end=stop \
--cuda-event-trace=false --cuda-memory-usage=false \
--force-overwrite=false --output=timeline ./existing-probe
Two captures without NVLOG and one subsequent capture with NVLOG yielded the same kernel/API counts, 112/100 totals and warnings. The final capture added global 100 NVLOG with ForceFlush; its timings are not used for performance comparison.
Observations
| Observation | Result |
|---|---|
| Expected launches / kernels | 32 / 32, unique matching correlation IDs, positive kernel intervals, same device/context/stream and serial order |
| Report API / kernel count | 68 / 32; sum 100 |
| Diagnostic totals | CUPTI produced 112; CUDA collected 100 |
| Completeness warning | Not all CUDA events might have been collected. |
| Other warning | Unified Memory cannot be traced on devices that do not support peer-to-peer transfers |
| Application-thread ProfilerStop | Current-context synchronization CUresult 0; CUPTI flush flags 0, CUptiResult 0 |
| Later tool/control-thread analysis stop | Current-context synchronization CUresult 201; CUPTI flush flags 1, CUptiResult 0 |
In the final NVLOG, raw CUDA transport records total 122: Context 1, Stream 8, Device 1, ContextBinding 1, Api 69, Overhead 7, Kernel 32 and Synchronization 3. All 32 report kernel IDs and all 68 report API IDs match raw records.
The only extra raw API has correlation 103, driver CBID 17, return 201 and the tool/control-thread ID; an associated synchronization record has ContextId 0. Based on alignment through 68 shared API timestamps, it appears after the capture end: approximately 65.693–66.504 ms versus a session stop at 60.112 ms. This is internal timestamp-representation alignment, not independent clock calibration.
Buffer-completion counts are 1+8 before capture, 104+3 at ProfilerStop and 3+2 at analysis stop. The latter groups sum to 112; 122 minus the 9 precapture records and one ContextBinding packet also equals 112. These are arithmetic reconciliations; we do not know the exact buffer-to-category mapping or internal counter definitions.
No explicit dropped-record count or warning-trigger branch was available in these logs. The warning is present in the report diagnostics but not verbatim in profile NVLOG. Its diagnostic timestampType differs from the producer diagnostic, so we have not used their numeric values to infer causal ordering.
Questions
- In 2026.5.1.161, which activity categories and capture phases are included in the produced and collected totals? Should synchronization, overhead, metadata and out-of-range auxiliary records affect both counters equally?
- What conditions generate this specific completeness warning? Can normal category/range filtering generate it, or does it imply another loss/incompleteness check? Which existing report/log fields distinguish these cases?
- Is analysis-stop synchronization returning CUDA_ERROR_INVALID_CONTEXT on a tool/control thread expected when that thread has no current context? Could it independently trigger this warning even when application ProfilerStop synchronization/flush and all expected kernels succeeded?
- If the existing counters are insufficient, what is the minimum additional diagnostic information needed to distinguish normal filtering from buffer drops or incomplete activity records? We would prefer version-specific log settings or an interpretation of existing data before repeating the same capture.
For context, the CUPTI 13.4 Activity API describes FlushAll as not implicitly synchronizing, forced flush as able to return incomplete records, and separate dropped-record APIs. The Nsight Diagnostics Summary documentation notes that some diagnostics are added during report analysis. Neither description establishes the cause of this particular warning.
Thank you.