Isaac SDK Error: Assertion Error in trtSmToCask

I tried to run the pick and place example application in a docker container (built using the steps given here: https://docs.nvidia.com/isaac/isaac/doc/getting_started.html#using-docker) using the following command:

bazel run //apps/samples/pick_and_place – --arm ur10

I got the following error:

2021-08-06 22:44:54.406 WARN  external/com_nvidia_isaac_engine/engine/alice/backend/codelet_canister.cpp@229: Codelet '_engine_tcp_udp_asio_context/isaac.alice.AsioContext' was not added to scheduler because no tick method is specified.
2021-08-06 22:44:54.406 INFO  packages/sight/WebsightServer.cpp@247: Sight webserver is loaded
2021-08-06 22:44:54.406 INFO  packages/sight/WebsightServer.cpp@248: Please open Chrome Browser and navigate to http://<ip address>:3000
2021-08-06 22:44:54.406 WARN  external/com_nvidia_isaac_engine/engine/alice/backend/codelet_canister.cpp@229: Codelet 'websight/isaac.sight.AliceSight' was not added to scheduler because no tick method is specified.
2021-08-06 22:44:54.407 WARN  external/com_nvidia_isaac_engine/engine/alice/backend/codelet_canister.cpp@229: Codelet 'controller.kinematic_tree/KinematicTree' was not added to scheduler because no tick method is specified.
2021-08-06 22:44:54.407 DEBUG external/com_nvidia_isaac_engine/engine/alice/backend/event_manager.cpp@40: Stopping node 'pose_initializer' because it reached status 'SUCCESS'
2021-08-06 22:44:54.407 WARN  external/com_nvidia_isaac_engine/engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick parameter
2021-08-06 22:44:54.407 WARN  external/com_nvidia_isaac_engine/engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick parameter
2021-08-06 22:44:54.417 WARN  external/com_nvidia_isaac_engine/engine/alice/backend/codelet_canister.cpp@229: Codelet '_check_operating_system/isaac.alice.CheckOperatingSystem' was not added to scheduler because no tick method is specified.
2021-08-06 22:44:54.417 WARN  external/com_nvidia_isaac_engine/engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick parameter
2021-08-06 22:44:54.418 WARN  external/com_nvidia_isaac_engine/engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick parameter
2021-08-06 22:44:54.424 WARN  packages/ml/TensorRTInference.cpp@463: Could not read from .plan (path is set to 'external/sortbot_pose_estimation_models/resnet18_detector_kltSmall.plan'). Falling back to building the .plan from frozen model external/sortbot_pose_estimation_models/resnet18_detector_kltSmall.etlt. Note: this process may take up to several minutes.
2021-08-06 22:45:04.403 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/allocator_backend.cpp@57: Optimized memory CPU allocator.
2021-08-06 22:45:04.403 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/allocator_backend.cpp@66: Optimized memory CUDA allocator.
2021-08-06 22:46:10.778 WARN  packages/ml/TensorRTInference.cpp@174: TRT WARNING: Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
2021-08-06 22:47:55.626 ERROR packages/ml/TensorRTInference.cpp@171: TRT ERROR: ../rtSafe/cuda/caskUtils.cpp (98) - Assertion Error in trtSmToCask: 0 (Unsupported SM.)
2021-08-06 22:47:55.626 PANIC packages/ml/TensorRTInference.cpp@258: Failed to build TensorRT engine from external/sortbot_pose_estimation_models/resnet18_detector_kltSmall.etlt.

I found out that there is no TensorRT installed and I thought that if I install TensorRT, the error will go away. I was wrong. Then, I found out that for the graphics processor I have, which is NVIDIA GeForce RTX 3080 Laptop GPU, I must install CUDA 11.1 and TensorRT 7.2.1 (as per the solution here: RTX 3070 / TensorRT Internal Error: Assertion failed: Unsupported SM and here: 3080 support · Issue #800 · NVIDIA/TensorRT · GitHub). After a lot of struggle over the course of a few days, I finally managed to obtain exactly these versions of CUDA and TensorRT in a docker container along with the other dependencies required for working with Isaac SDK (I used this command to install other dependencies: engine/engine/build/scripts/install_dependencies.sh --config=x86_64_cuda_11_1). No luck still. The error remains the same:

2021-08-06 23:44:33.986 WARN  packages/ml/TensorRTInference.cpp@463: Could not read from .plan (path is set to 'external/sortbot_pose_estimation_models/resnet18_detector_kltSmall.plan'). Falling back to building the .plan from frozen model external/sortbot_pose_estimation_models/resnet18_detector_kltSmall.etlt. Note: this process may take up to several minutes.
2021-08-06 23:44:43.936 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/allocator_backend.cpp@57: Optimized memory CPU allocator.
2021-08-06 23:44:43.936 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/allocator_backend.cpp@66: Optimized memory CUDA allocator.
2021-08-06 23:47:34.139 WARN  packages/ml/TensorRTInference.cpp@174: TRT WARNING: Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
2021-08-06 23:49:55.527 ERROR packages/ml/TensorRTInference.cpp@171: TRT ERROR: ../rtSafe/cuda/caskUtils.cpp (98) - Assertion Error in trtSmToCask: 0 (Unsupported SM.)
2021-08-06 23:49:55.527 PANIC packages/ml/TensorRTInference.cpp@258: Failed to build TensorRT engine from external/sortbot_pose_estimation_models/resnet18_detector_kltSmall.etlt.

For reference, the output of the python3 ../engine/engine/build/scripts/version_checker.py command in the docker container I have created looks like this:

---------------------------------------------------------------
|Package             |Recommended Version |Current Version     |
---------------------------------------------------------------
|OS                  |Ubuntu 18.04.2 LTS  |Ubuntu 18.04.5 LTS  |
|Bazel               |3.1.0               |3.1.0               |
|GPU_Driver          |>=440               |470.57.02           |
|Cuda                |10.2.x              |11.1.105            |
|Cudnn               |8.0.3.x             |8.0.5.39            |
|TensorRt            |7.1.5               |7.2.1.6             |
|TensorFlow          |1.15.0              |2.5.0               |
|pycapnp             |>=0.6.3             |0.6.4               |
|librosa             |>=0.6.3             |0.8.0               |
|SoundFile           |>=0.10.2            |0.10.3.post1        |
|Python2             |2.7.x               |2.7.17              |
|Python3             |3.6.x               |3.6.9               |
---------------------------------------------------------------

Environment

TensorRT Version : 7.2.1.6
GPU Type : RTX3080 Laptop GPU
Nvidia Driver Version : 470.57.02
CUDA Version : 11.1.1
CUDNN Version : 8.0.5.39
Operating System + Version : Ubuntu 18.04.5
Python Version (if applicable) : 3.6.9

Is there some way to find out what is causing this error because I am unable to find the file TensorRTInference.cpp on my system so that I can atleast print out what is wrong with the setup I have? Any help will be much appreciated.

1 Like

Probably the solution given here is the explanation of the issue and it cannot be resolved: Isaac SDK 2021.1 Jetbot Jupyter Notebook Inference Sim Example crashes - #3 by eric.chan

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.