I have compiled mobilenet model using AWS. Deployed my model to Jetson Nano Device using Greengrass. When I run inference application, it throws DLR error.
I have installed jetpack4.2.1 using SD Card Image. I can see Cuda10.0; TensorRt 5.1.6; CudNN 7 on my device.
Error StackTrace:
[2020-12-21T12:46:09.789+05:30][ERROR]-init.py:1028,2020-12-21 12:46:09,622 INFO Found libdlr.so in model artifact. Using dlr from /ml_model/libdlr.so
[2020-12-21T12:46:11.974+05:30][ERROR]-NvRmPrivGetChipIdLimited: Could not read Tegra chip id/rev
[2020-12-21T12:46:11.974+05:30][ERROR]-Expected on kernels without fuse support, using Tegra K1
[2020-12-21T12:46:11.974+05:30][ERROR]-NvRmPrivGetChipPlatform: Could not read platform information
[2020-12-21T12:46:11.974+05:30][ERROR]-Expected on kernels without fuse support, using silicon
[2020-12-21T12:46:11.975+05:30][ERROR]-libnvrm_gpu.so: NvRmGpuLibOpen failed
[2020-12-21T12:46:11.984+05:30][ERROR]-[12:46:11] /home/nvidia/neo-ai-dlr/src/dlr.cc:327: [12:46:11] /home/nvidia/neo-ai-dlr/3rdparty/tvm/src/runtime/cuda/cuda_device_api.cc:115: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: unknown error
[2020-12-21T12:46:11.984+05:30][ERROR]-Stack trace:
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (0) /ml_model/libdlr.so(+0x531b00) [0x7f8cd92b00]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (1) /ml_model/libdlr.so(tvm::runtime::CUDADeviceAPI::AllocDataSpace(DLContext, unsigned long, unsigned long, DLDataType)+0x178) [0x7f8cd97a28]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (2) /ml_model/libdlr.so(+0x13100b0) [0x7f8db710b0]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (3) /ml_model/libdlr.so(+0x13489a8) [0x7f8dba99a8]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (4) /ml_model/libdlr.so(+0x134b528) [0x7f8dbac528]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (5) /ml_model/libdlr.so(dlr::TVMModel::SetupTVMModule(std::vector<ModelElem, std::allocator > const&)+0x3bc) [0x7f8cd8c2a4]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (6) /ml_model/libdlr.so(dlr::TVMModel::SetupTVMModule(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&)+0x174) [0x7f8cd8e244]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (7) /ml_model/libdlr.so(CreateDLRModel+0x2ec) [0x7f8cd40fec]
[2020-12-21T12:46:11.984+05:30][ERROR]- [bt] (8) /usr/lib/aarch64-linux-gnu/libffi.so.6(ffi_call_SYSV+0x64) [0x7f90fb6d28]
[2020-12-21T12:46:12.018+05:30][ERROR]-init.py:1028,2020-12-21 12:46:11,984 ERROR error in DLRModel instantiation
[2020-12-21T12:46:12.018+05:30][ERROR]-Traceback (most recent call last):
[2020-12-21T12:46:12.018+05:30][ERROR]- File “/usr/local/lib/python3.7/dist-packages/dlr/api.py”, line 85, in init
[2020-12-21T12:46:12.018+05:30][ERROR]- self._impl = DLRModelImpl(model_path, dev_type, dev_id, error_log_file, use_default_dlr)
[2020-12-21T12:46:12.018+05:30][ERROR]- File “/usr/local/lib/python3.7/dist-packages/dlr/dlr_model.py”, line 82, in init
[2020-12-21T12:46:12.018+05:30][ERROR]- c_int(dev_id)))
[2020-12-21T12:46:12.018+05:30][ERROR]- File “/usr/local/lib/python3.7/dist-packages/dlr/dlr_model.py”, line 160, in _check_call
[2020-12-21T12:46:12.018+05:30][ERROR]- raise DLRError(self._lib.DLRGetLastError().decode(‘ascii’))
[2020-12-21T12:46:12.018+05:30][ERROR]-dlr.dlr_model.DLRError
[2020-12-21T12:46:12.018+05:30][FATAL]-lambda_runtime.py:142,Failed to import handler function “lambda_function.lambda_handler” due to exception:
[2020-12-21T12:46:12.018+05:30][FATAL]-lambda_runtime.py:382,Failed to initialize Lambda runtime due to exception:
I am not what causes this issue. If you need any further details, please let me know.
Any hint would be appreciable.