Hello everyone,
I am using the Jetson Nano development kit for implementing computer vision on an autonomous robotic platform. I have a setup of multiple RealSense cameras (one for each side of the platform) and am implementing a segmentation model. The project runs successfully when I use my laptop (through it doesn’t have any GPU cores) but fails when I run it on the Jetson. The project is successful only on 1 camera, but when I send a UDP request to get the image from another camera attached to the Jetson, it fails showing a memory access error.
I have taken screenshots of the GPU stats (which show high memory usage), as well the error pasted below. It would be great if I could get suggestions on hardware/memory optimizations I could include so that it runs smoothly on a multi-camera setup using the Jetson.
Error:
Receiving images from camera 11
2
2025-08-21 11:03:37.602821438 [E:onnxruntime:Default, cuda_call.cc:123 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=65 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
2025-08-21 11:03:37.603099485 [E:onnxruntime:Default, cuda_call.cc:123 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_));
Traceback (most recent call last):
File “/home/metazet/path_detection_jetson/software4metazet/run_segmentation_BART.py”, line 1592, in
main()
File “/home/metazet/path_detection_jetson/software4metazet/run_segmentation_BART.py”, line 1584, in main
model.run_deploy()
File “/home/metazet/path_detection_jetson/software4metazet/run_segmentation_BART.py”, line 1196, in run_deploy
interface.handle_udp_requests()
File “/home/metazet/path_detection_jetson/software4metazet/communication.py”, line 173, in handle_udp_requests
result = self.run_row_detection_callback(actual_cam_id)
File “/home/metazet/path_detection_jetson/software4metazet/run_segmentation_BART.py”, line 209, in run_row_detection
result = self.run_inference(bgr_img, depth_img, key)
File “/home/metazet/path_detection_jetson/software4metazet/run_segmentation_BART.py”, line 313, in run_inference
self.run_model()
File “/home/metazet/path_detection_jetson/software4metazet/run_segmentation_BART.py”, line 373, in run_model
output = self.inferencer.process_image(self.bgr_img)
File “/home/metazet/path_detection_jetson/software4metazet/smp_default_model_run.py”, line 66, in process_image
result = self.ort_session.run(
File “/home/metazet/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py”, line 273, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=65 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
terminate called after throwing an instance of ‘onnxruntime::OnnxRuntimeException’
what(): /opt/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:129 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; SUCCTYPE = cudaError; std::conditional_t<THRW, void, onnxruntime::common::Status> = void] /opt/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:121 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; SUCCTYPE = cudaError; std::conditional_t<THRW, void, onnxruntime::common::Status> = void] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/cuda_allocator.cc ; line=98 ; expr=cudaFreeHost(p);
Aborted (core dumped)
