[NVTX] Export nvtx events from nsys

[puneethnaik]

I have annotated the to be profiled python code with nvtx ranges supported by pytorch. Here is the screenshot of one of them:

I would like to export them for some post processing. I would also like the nested structure to be maintained. When I just copy the events using Ctrl+C, I lose the nested structure.
Question
Is there any way to export nvtx ranges shown in the events view to a format for post processing while maintaining the nested structure of nvtx ranges?

Thanks

[hwilper]

@jkreibich do we have a way of preserving NVTX nesting into the sqlite? I do not believe so, but you are the expert.

[jkreibich]

To my knowledge, the NVTX push/pop records don’t actually contain data about the range “stack” or what the “parent” enclosing range is. That is certainly true by the time they’re exported into SQLite.

I believe that anytime the ranges are displayed in hierarchical way, the nesting was recomputed on the fly. This is done by filtering the ranges by domain, PID, and TID, and then looking at the start/end times to find the “nearest” enclosing range. I know this is the way Nsight “stats” does it.

As an example, see nsys stats --report nvtxpptrace <input file> on how to do this. You can find the script source in <nsys install dir>/reports/nvtxpptrace.py. Stats uses the “R-tree” feature of SQLite to build a dimensional index to make filtering and correlation more efficient, but the basic process is as-described.

[puneethnaik]

   Start (ns)       End (ns)     Duration (ns)  DurChild (ns)  DurNonChild (ns)                         Name                            PID      TID    Lvl  NumChild  RangeId  ParentId  RangeStack                       NameTree                      
 --------------  --------------  -------------  -------------  ----------------  ---------------------------------------------------  -------  -------  ---  --------  -------  --------  ----------  ---------------------------------------------------
 13,456,656,229  13,462,758,642      6,102,413              0         6,102,413  Region 1                                             150,219  150,219    0         0        1            :1          Region 1                                           
 14,542,903,861  14,543,115,685        211,824        202,537             9,287  aten::to, op_id = 1                                  150,219  150,219    0         1        2            :2          aten::to, op_id = 1                                
 14,542,912,774  14,543,115,311        202,537        185,023            17,514  aten::_to_copy, op_id = 2                            150,219  150,219    1         2        3         2  :2:3        --aten::_to_copy, op_id = 2                        
 14,542,926,266  14,543,073,791        147,525              0           147,525  aten::empty_strided, op_id = 3                       150,219  150,219    2         0        4         3  :2:3:4      ----aten::empty_strided, op_id = 3                 
 14,543,077,036  14,543,114,534         37,498              0            37,498  aten::copy_, op_id = 4                               150,219  150,219    2         0        5         3  :2:3:5      ----aten::copy_, op_id = 4                         
 14,543,166,289  14,572,018,929     28,852,640     28,850,174             2,466  aten::to, op_id = 5                                  150,219  150,219    0         1        6            :6          aten::to, op_id = 5                                
 14,543,167,747  14,572,017,921     28,850,174     28,839,916            10,258  aten::_to_copy, op_id = 6                            150,219  150,219    1         2        7         6  :6:7        --aten::_to_copy, op_id = 6                        
 14,543,171,058  14,543,440,984        269,926              0           269,926  aten::empty_strided, op_id = 7                       150,219  150,219    2         0        8         7  :6:7:8      ----aten::empty_strided, op_id = 7                 
 14,543,443,396  14,572,013,386     28,569,990              0        28,569,990  aten::copy_, op_id = 8                               150,219  150,219    2         0        9         7  :6:7:9      ----aten::copy_, op_id = 8                         
 14,572,058,541  14,572,061,218          2,677              0             2,677  aten::_has_compatible_shallow_copy_type, op_id = 9   150,219  150,219    0         0       10            :10         aten::_has_compatible_shallow_copy_type, op_id = 9 
 14,572,068,102  14,572,068,403            301              0               301  aten::_has_compatible_shallow_copy_type, op_id = 10  150,219  150,219    0         0       11            :11         aten::_has_compatible_shallow_copy_type, op_id = 10

Thanks, jkreibich for the insight. This solves the problem. For people who will visit in the future, I have enclosed a few entries of the result of the command suggested by jkreibich. We can see the nesting structure in NameTree field of the output. Each level of nesting is represented by two dashes. Thanks again for the help @hwilper @jkreibich