AIAA engine doesn't work properly

Hi,

I am using Clara Train SDK with TRITON engine without problems. When I change engine to AIAA, I am able to run containers, download and upload models to the server, but when I do auto-segmentation in 3D Slicer, I obtain the below mention error message. Do you have any idea why? The weird thing is that Triton engine is working. I am using original model clara_pt_liver_and_tumor_segmentation without any change. Thanks for your help

AIAA Exception
Traceback (most recent call last):
File “/mnt/UbuntuData/vavr/Tools/Slicer-4.11.20210226-linux-amd64/NA-MIC/Extensions-29738/NvidiaAIAssistedAnnotation/lib/Slicer-4.11/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py”, line 410, in onClickSegmentation
extreme_points, result_file = self.logic.segmentation(in_file, session_id, model)
File “/mnt/UbuntuData/vavr/Tools/Slicer-4.11.20210226-linux-amd64/NA-MIC/Extensions-29738/NvidiaAIAssistedAnnotation/lib/Slicer-4.11/qt-scripted-modules/SegmentEditorNvidiaAIAALib/SegmentEditorEffect.py”, line 1092, in segmentation
session_id=session_id,
File “/mnt/UbuntuData/vavr/Tools/Slicer-4.11.20210226-linux-amd64/NA-MIC/Extensions-29738/NvidiaAIAssistedAnnotation/lib/Slicer-4.11/qt-scripted-modules/NvidiaAIAAClientAPI/client_api.py”, line 389, in inference
raise AIAAException(AIAAError.SERVER_ERROR, ‘Status: {}; Response: {}’.format(status, form))
NvidiaAIAAClientAPI.client_api.AIAAException: (3, ‘Status: 500; Response: b’{“error”:{“message”:[“No CUDA GPUs are available”],“type”:“RuntimeError”},“success”:false}\n’’)
Run Segmentation for model: clara_pt_liver_and_tumor_ct_segmentation_v1 - Status: 500; Response: b’{“error”:{“message”:[“No CUDA GPUs are available”],“type”:“RuntimeError”},“success”:false}\n’
Run segmentation for (clara_pt_liver_and_tumor_ct_segmentation_v1): FAILED

This error means the container can not see any GPUs.

We recommend using the Triton engine.

If you really want to use AIAA engine (native engine), then when starting that container you need to specify GPU options there.

If you are following Getting Started — Clara Train SDK v4.0 documentation
You need to make following modifications for your docker-compose.yml.
You need to have the following section under clara-train-sdk service (let it see the GPU)

    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]

And you can safely remove all the part under tritonserver service.

Do you have any idea where could be the problem if I have two identical computers and for one of them the segmentation in 3D Slicer is working with original docker-compose.yml? And for the second computer I need to do modifications and specify GPU?

By identical, do you mean their CPU and GPU are both the same?

What about the OS version/3D Slicer version?

Do you remember to create a workspace on both computers and specified them in docker-compose.yml?

If all these are the same, and these computers only running AIAA, then I think they should work the same way.