Is it possible to run a Triton inference server with DALI backend on AGX Xavier running JP4.6

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
,
1

I have a Jetson AGX Xavier with JP4.6 that I cannot upgrade further as I have incompatible with JP5 camera drivers. I would like to run Triton on that Jetson as a local server and I want to use DALI+TRT ensambles.

I am however facing the following problem. DALI has become available by default in Triton since version 20.11 which is more than 2 years ago, however according to the official Triton release notes, it requires CUDA 11.1

image
image753×74 4.07 KB

And JetPack 4.6 comes with CUDA 10.2.

I tried to run a docker image on the Jetson of a more up-to-date triton container, 22.10, and added a simple Dali model to serve as a test and I get the following error:

Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File "<frozen importlib._bootstrap>", line 553, in module_from_spec
AttributeError: 'NoneType' object has no attribute 'loader'
Traceback (most recent call last):
  File "<string>", line 8, in <module>
  File "/opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.8/site-packages/nvidia/dali/_utils/autoserialize.py", line 70, in invoke_autoserialize
    dali_pipeline().serialize(filename=filename)
  File "/opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 1166, in serialize
    self._init_pipeline_backend()
  File "/opt/tritonserver/backends/dali/conda/envs/dalienv/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 693, in _init_pipeline_backend
    self._pipe = b.Pipeline(self._max_batch_size,
RuntimeError: CUDA runtime API error cudaErrorInsufficientDriver (35):
CUDA driver version is insufficient for CUDA runtime version

So my question is, has anyone done this before? Can I run DALI backed triton server on JP4.6? I’d love to hear your feedback before I waste a dozen hours over something that could be bound to fail.

3

Hi @alexander.soklev, that container isn’t built for Jetson/JetPack, so it won’t work as intended. Instead you could try one of the deepstream-l4t:*-triton containers that’s compatible with the version of JetPack-L4T that you are running.

I’m uncertain if those containers were built with the DALI backend however, and I’m not personally experienced with building it, but I was able to find this about building it for aarch64:

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.