- 我们期望在一段连续的工作状态下, 由不同场景通过mps 限制不同的算力 来让模型帧率和延时 进一步得到动态控制 ,但是 发现模型 在onnx 转成.engine 模型时,会受到SM总个数的影响,真实推理时,通过mps 限制时,模型获得的sm 总数不一样, 会导致 警告以及如下错误: 1787128493.075111 tegra-ubuntu-pos mirobot_sixdof_start.sh[53069]: [prc_surp_one-1] [FS-TRT] IExecutionContext::enqueueV3: Error Code 1: Myelin
([exec_instruction.cpp:exec:1190] CUDA error 720 launching __myl_AddMulMaxReluMeanMaxr kernel. In executeMyelinGraph at runtime/myelin/
… +28 lines (ctrl + t to view transcript)
([exec_instruction.cpp:exec:1190] CUDA error 720 launching __myl_AddMulMaxReluMeanMaxr kernel. In executeMyelinGraph at runtime/myelin/
runner.cpp:778) - 通过mps 控制使用时, 我们通过nvidia-smi 或者cupti 获得的GPU使用率,包括SM使用率都是100,不会随着mps 设置40%, GPU使用率变成40%, 从观测手段,是否有新的观测 ,除了在算法内,打出能看到的SM数(变成8个SM)
Hi,
Have you tried to convert the engine with MPS enabled so the resources between compiling and runtime are aligned?
If you are using JetPack 7.2+, could you try MIG, which is a hardware-based slice?
Thanks.
目前也只能是这么做,就是编译时 和运行时 ,都给他60%(也就是12个SM) , 但是这样会引生出来另一个问题,如果 我们改变了 模型的资源分配,所有的.engine 都需要对应的资源全部重新编译,太麻烦了.
我这边想到唯一的解法, 都按照最小单元4个sm 资源去编译, 运行时,向上兼容, 但是这样会引出另外一个问题,就是效率有点慢.
你们设计mps 时候, 有考虑到这一点吗?
JetPack 7.2 目前还在验证阶段,还没有完全合入主分支. 另外MIG 及时功能好用,我理解他从原理上,也不能完全规避这种风险吧,因为MIG 切片也是再切SM , 模型和这个SM 个数也没有剥离开呀
Hi,
TensorRT is optimized based on the resources so it’s expected to compile a different engine when the SM number changes.
Maybe you can try to convert the engine for all possible profiles (ex. 12 SM) in the beginning.
And then load the engine accordingly when the resources change.
Thanks.
Hello,
I would like to consult whether there is relevant technology available for the Jetson platform. I came across a document on the NVIDIA official website introducing an adaptive inference engine, which appears to address the pain point that the TensorRT inference framework requires compiling and generating multiple .engine files under MPS multi‑process resource‑allocation scenarios. According to the document, it delivers decent performance. Could you please confirm whether this adaptive inference engine is supported on the Jetson‑Thor platform?
Hi,
The feature is from TensorRT for RTX, which requires an RTX GPU:
Thanks.