Query Related to NVIDIA DevTools Sidecar Injector

Developer Tools Nsight Systems Profiling Linux Targets
1

Hi Team,
I am going through NVIDIA DevTools Sidecar Injector | NVIDIA NGC for using Nvidia profiling for containerized applications in K8s since we cannot modify application start parameters to add nsys profile in the prefix.

Does it support application which gets started using some script in the K8s pod command ? If the injector modifies command for the main application by adding nsys profile as prefix then how to make it work for cases where application started via some script present in the command.

Similarly is there way to test the side car that gets added by the injector via without deploying the injector and we directly modifying the pod yaml spec to add the Nvidia dev tool side car ? is there a way for that.

2

@mpopov can you comment on this?

3

Hi @jaiwant23,

Sure, The NVIDIA DevTools Sidecar Injector can inject Nsight Systems into any application(s) inside a Pod, not just the main application. To select the desired application(s), specify a regex that identifies the application(s) in the injectionMatch configuration value.

Regarding the second question. I am not quite sure I fully understand your question. Could you explain a bit more about what you’re trying to accomplish with this configuration? You could try using the NVIDIA Nsight Systems CLI Docker Container directly and try to configure everything yourself in the Yaml file, but I would strongly recommend using the NVIDIA DevTools Sidecar Injector. The Injector installation automatically configures everything needed for profiling, and after its uninstallation and the Pod(s) restart, there should be no traces of the NVIDIA DevTools Sidecar Injector’s presence.

4

Hi @mpopov

Thanks for the details I have some further queries based on your answers

For 1)

  • Confirming following if there are 2 containers running into a POD one is main container and other side container, we can do profiling on both of the containers application via the regex or any of them also via regex. The injectionMatch property seems to be at injector level if new applications comes and they also need this profiling supported then they will need to be whitelisted in this regex every time right for new application ? Is it possible to able to specify this regex at Pod level so we can control application at pod only instead at global level that is currently present at injector level.
  • There is no restriction on how the application is started in the container of the pod right ? like it can be via some script or it can be as a simple as javac application for a java application.

For 2)
I was asking this for checking the NVIDIA DevTools Sidecar and currently it seems the Injector needs to be deployed for side car injection which will take time so I want to check above tool quickly.

5

Thank you @jaiwant23 for the provided clarification.

  • Confirming following if there are 2 containers running into a POD one is main container and other side container, we can do profiling on both of the containers application via the regex or any of them also via regex. The injectionMatch property seems to be at injector level if new applications comes and they also need this profiling supported then they will need to be whitelisted in this regex every time right for new application ? Is it possible to able to specify this regex at Pod level so we can control application at pod only instead at global level that is currently present at injector level.

Currently, we have the option to configure the injectionMatch for each namespace, as described in the Separate Configurations section of the documentation. Additionally, you can label only the Pods required for profiling. Meanwhile, we are already working on providing more granular control over the injection.

  • There is no restriction on how the application is started in the container of the pod right ? like it can be via some script or it can be as a simple as javac application for a java application.

Yes, there is no restriction on that.

I was asking this for checking the NVIDIA DevTools Sidecar and currently it seems the Injector needs to be deployed for side car injection which will take time so I want to check above tool quickly.

I am afraid to test the tool you need to deploy it, but:

  • It should be fast if you’re concerned about deployment time.
  • If you did not label any Pods or namespaces, the Injector will not inject or modify any resources except for the internal tool’s resources.
  • You can test it on a separate test Pod.
1 Like