Need help setting up Nsight Systems to remotely profile Jetson Nano (Ubuntu 19.10)

I’m having a hard time finding instructions on doing the setup like that properly.
I used to have ubuntu 18.10 and I managed to grab the arm target header files from android code works.
It somewhat worked, but was not capturing the cuda traces.

What’s the proper way to get those headers in 19.10?

The docs say to get cuda-repo-cross-_all.deb
but I have no idea where to get that.

the cuda installer has the url for the x84 host, which I used to install cuda 10.2:
cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01,

but I don’t understand where do I get the repo files for arm architecture.

I understand that 19.10 is not supported, but I cant roll back to 18.
The local cuda profiler seems to work beautifully despite the version, so I just want to try to see if it’s going to work with jetson.

And in general, I’d like to understand the cross-platfrom cuda installation process.

If someone could point me in the right direction, that would be awesome.

Thanks.

Hi,

I’m afraid that Nsight system may not work on Ubuntu19.10.
But it’s still recommended to give it a try first.

It looks like CUDA doesn’t work on your environment yet.
Would you mind to check this tutorial to get CUDA working at first.
https://devtalk.nvidia.com/default/topic/1051465/linux/ubuntu-19-04-disco-dingo/

After that, you should be able to remote access the Jetson with Nsight system.
However, we don’t expect cross-compile can work but remote compiling.

Thanks.

Thank you. Yes, I’ve given up on my idea to try and make it work and just installed 18.04 on a separate drive to make sure things are fully compatible.

I still coulnd’t get the arm target headers files via dpkg, so I got them from Codeworks again.
(I assumed that if I do add-architecture aarch64 and do apt update, I should be able to access the cross-compilation package for arm, but apt just starts spewing 404 errors and nothings gets updates, so not sure what to do here)

But anyways, The remote connection works. However, there’s a problem.
I can only get the samples pobably 1 out of 5 times.

Most of the times the app stops after first couple hundred samples, and the logs just say ‘operation aborted’. But occasionally it goes through this point and starts gathering samples, and in the end I’m able to download the trace and view everything normally.

So it does work, but I cant figure out why it stops. Do you think it’s something that can be fixed in the profiler itself, or there’s probably something on the Jetson that’s stopping the gather?

Hi,

Are you install the Nsight tool from JetPack?
If not, it’s recommended to install the package from JetPack first.
https://developer.nvidia.com/embedded/jetpack

Thanks.

Ok, so I wiped my dev machine, installed ubuntu 18.04 and jetpack package.
profining worked first try, no issues.

However, the problem still remains.
First, I don’t seem to be able to get any cuda samples. (and I checked the ‘run as root’ checkbox, the messages say that cuda injection is successful)
Second, most of the time I try to profile - it fails and disconnects from ssh. The messages say

Event requestor failed: Source ID=
Type=ErrorInformation (18)
 Properties:
  OriginalSource (145)=EventRequestor
 Error information:
 InternalError (105)
  Properties:
  ErrorText (100)=The operation was aborted.
  ServiceName (200)=AnalysisService
  MethodName (201)=GetData

and

Events fetch failed: Source ID=
Type=ErrorInformation (18)
 Error information:
 ProcessEventsError (4005)
  Properties:
  ErrorText (100)=/build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/SymbolAnalyzer/SymbolAnalyzer.cpp(326): Throw in function void QuadDSymbolAnalyzer::SymbolAnalyzer::HandleKernelSymbols(const QuadDTimestamp&, uint64_t, uint64_t, const boost::filesystem::path&, QuadDSymbolAnalyzer::ISymbolResolver::Continuator, QuadDSymbolAnalyzer::StateMap*, const Ptr&)
Dynamic exception type: boost::exception_detail::clone_impl
std::exception::what: GetKernelSymbolsError
[QuadDCommon::tag_error_text*] = /build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/SymbolAnalyzer/SymbolAnalyzer.cpp(326): The operation was aborted.

It always goes like this: I launch the app, and my profiling always is set to 5second delay.
As soon as prifiling starts, it either gets couple hundred or a few thousand samples, and then dies.

I have to reconnect to shh, restart the profiler. about 1 out of 10 times it works ok.
Sometimes it works like 5 times in a row.

It doesnt seem to be connected to anything my app is doing. And I tried rebooting Jetson a lot, I also restart nvargus service each time manually (the app is foing capture from camera)

Hi,

Sorry for the late update.

It looks like you are running into the similar issue of this topic:
http://142.93.83.82/t/using-qdstrmimporter/1779/2

Would you mind to check if the comment also helps for you?
Thanks.

Sorry to bother you again,

But the manual import tool gives me this:

Importing [0%                                                                 ]
Importing [=============25%                                                   ]Import Failed with unexpected exception: /build/agent/work/20a3cfcd1c25021d/QuadD/Host/QdstrmImporter/main.cpp(60): Throw in function bool {anonymous}::RunImport(const boost::filesystem::path&, const boost::filesystem::path&)
Dynamic exception type: boost::wrapexcept<QuadDCommon::RuntimeException>
std::exception::what: RuntimeException
[QuadDCommon::tag_error_text*] = Status: AnalysisFailed
Error {
  Type: RuntimeError
  SubError {
    Type: InvalidArgument
    Props {
      Items {
        Type: OriginalExceptionClass
        Value: "N5boost16exception_detail10clone_implIN11QuadDCommon24InvalidArgumentExceptionEEE"
      }
      Items {
        Type: OriginalFile
        Value: "/build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/Clients/AnalysisHelper/AnalysisStatus.cpp"
      }
      Items {
        Type: OriginalLine
        Value: "168"
      }
      Items {
        Type: OriginalFunction
        Value: "QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusInfo QuadDAnalysis::AnalysisHelper::AnalysisStatus::MakeEventStatus(const QuadDAnalysis::EventSourceStatus&, QuadDCommon::GlobalVm)"
      }
    }
  }
}

The version I’m using is “NVIDIA Nsight Systems version 2019.6.2.6-3ffb807”

It worked couple times when my app crashed after a few seconds. But when I capture the actual thing that I need - it doesn’t work.

The symbol processing timeout makes sense because my Jetson board is in another city (ping 30ms). At the same time it makes no sense, because the trace files are < 1mb.

I’ve reported this as a bug via nsight systems report tool.

At this point I think I’ll look into another profiling tool. Too many things are going wrong.

Thank you very much for you help.