This past month I have been using Triton and absolutely love it. On my local machine, everything works as expected. However, when I transferred the scripts and containers to our deployment machine, I keep getting hit with this error as soon as the client connects.
E0517 13:08:01.254093 1 logging.cc:40] Assertion failed: status == 0 /home/jenkins/workspace/OSS/L0_MergeRequest/oss/plugin/common/kernels/proposalKernel.cu:703 Aborting...
Before this error, the server is functional and loads every model properly. It waits there until being called by the client with no issue at all. However, as soon as the client calls the server over grpc this error occurs.
Please advise on how to fix this issue as I have been trying to debug it for almost 2 weeks now with no luck.
The server is using 4 p100 gpus.