Cuda Driver (an illegal memory access was encountered)

• Hardware Platform (Jetson / GPU)
Jetson AGX Xavier
• DeepStream Version
6.3
• JetPack Version (valid for Jetson only)
R35
• TensorRT Version
8.5.2-1
• NVIDIA GPU Driver Version (valid for GPU only)
CUDA 11.4
• Issue Type( questions, new requirements, bugs)
Question/bug

I am randomly seeing this error:

ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)

It seems to resolve itself when restarting the application. What causes it? How can it be mitigated to avoid these restarts?

We also think it may the cause or symptom of some more catastrophic errors we are seeing where the disk is put into a read-only mode. This requires a full AGX reboot which is causing some very big problems for us.

Which sample can be used to reproduce this problem?

Based on the provided context,I can’t tell where the problem is.

We were able to narrow down the problem to the 64GB Xavier AGX with disk encryption. Our 32GB models do not have the issue, regardless of encryption, nor does a 64GB model without encryption. The problem can be replicated by running the deepstream-app in sample_apps on an mp4 video. It doesn’t happen everytime, but we see the error below when stopping the app mid video. This reflects the same error that occurs during normal operation and causes the disk to be corrupted and go into read-only mode. Let me know if you’d like me to share configs, TRT engine, video file, etc.

0:00:09.896022857 124672 0xaaab05183d80 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/etc/ptz/model.trt
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT image           3x1088x1920     
1   OUTPUT kFLOAT boxes           42840x4         
2   OUTPUT kFLOAT scores          42840x1         
3   OUTPUT kFLOAT labels          42840x1         

0:00:09.975118898 124672 0xaaab05183d80 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /etc/ptz/model.trt
0:00:10.622316895 124672 0xaaab05183d80 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/root/deepstream_inference/configs/config_infer_primary.txt sucessfully
0:00:10.626827573 124672 0xaaab05183d80 WARN                 basesrc gstbasesrc.c:3600:gst_base_src_start_complete:<source> pad not activated yet
0:00:10.628931977 124672 0xaaab05183d80 WARN                 basesrc gstbasesrc.c:3600:gst_base_src_start_complete:<source> pad not activated yet

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume


**PERF:  FPS 0 (Avg)	
**PERF:  0.00 (0.00)	
** INFO: <bus_callback:239>: Pipeline ready

0:00:10.653581991 124672 0xaaab051765e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type tmcd
0:00:10.653851282 124672 0xaaab051765e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type pasp
0:00:10.654106748 124672 0xaaab051765e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type sgpd
0:00:10.654186240 124672 0xaaab051765e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type sbgp
0:00:10.654265507 124672 0xaaab051765e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type nmhd
0:00:10.654338630 124672 0xaaab051765e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type tmcd
0:00:10.654916541 124672 0xaaab051765e0 WARN                 qtdemux qtdemux.c:3250:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:10.655505460 124672 0xaaab051765e0 WARN                 qtdemux qtdemux.c:3250:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
0:00:10.655911269 124672 0xaaab051765e0 WARN                 qtdemux qtdemux.c:3250:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 3
Opening in BLOCKING MODE 
0:00:10.883938508 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:4561:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:10.884084914 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
0:00:10.884244505 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
0:00:10.884358397 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
0:00:10.884469634 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:11.021147412 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:4561:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:11.021295386 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
0:00:11.021425215 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
0:00:11.021533571 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
0:00:11.021677545 124672 0xfffeec051f00 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffee401e6a0 Failed to determine interlace mode
** INFO: <bus_callback:225>: Pipeline running

0:00:11.033109972 124672 0xfffeec051f00 WARN            v4l2videodec gstv4l2videodec.c:2305:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
0:00:11.043439443 124672 0xfffeec051f00 WARN          v4l2bufferpool gstv4l2bufferpool.c:1114:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:11.103246102 124672 0xfffef000b2a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1565:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
**PERF:  19.09 (18.64)	
**PERF:  19.28 (19.21)	
**PERF:  19.28 (19.20)	
**PERF:  19.28 (19.26)	
**PERF:  19.28 (19.25)	
**PERF:  19.28 (19.24)	
**PERF:  19.28 (19.26)	
**PERF:  19.28 (19.25)	
0:00:51.901802310 124672 0xfffef000a8c0 WARN            audiodecoder gstaudiodecoder.c:1753:gst_audio_decoder_drain:<faad0> still 1 frames left after draining
nvstreammux: Successfully handled EOS for source_id=0
** INFO: <bus_callback:262>: Received EOS. Exiting ...

Quitting
0:00:53.851773430 124672 0xaaab05183d80 ERROR                GST_BUS gstbus.c:1066:gst_bus_remove_watch:<bus1> no bus watch was present
App run successful
root@argus-wingtip:~/deepstream_inference/configs# timeout 30 /opt/nvidia/deepstream/deepstream-6.3/sources/apps/sample_apps/deepstream-app/deepstream-app -c deepstream_app_config.txt 
0:00:09.985589049 125225 0xaaaaf8e49d80 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/etc/ptz/model.trt
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT image           3x1088x1920     
1   OUTPUT kFLOAT boxes           42840x4         
2   OUTPUT kFLOAT scores          42840x1         
3   OUTPUT kFLOAT labels          42840x1         

0:00:10.067336461 125225 0xaaaaf8e49d80 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /etc/ptz/model.trt
0:00:10.721654970 125225 0xaaaaf8e49d80 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/root/deepstream_inference/configs/config_infer_primary.txt sucessfully
0:00:10.726105100 125225 0xaaaaf8e49d80 WARN                 basesrc gstbasesrc.c:3600:gst_base_src_start_complete:<source> pad not activated yet
0:00:10.728250403 125225 0xaaaaf8e49d80 WARN                 basesrc gstbasesrc.c:3600:gst_base_src_start_complete:<source> pad not activated yet

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume


**PERF:  FPS 0 (Avg)	
**PERF:  0.00 (0.00)	
** INFO: <bus_callback:239>: Pipeline ready

0:00:10.752804189 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type tmcd
0:00:10.753029638 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type pasp
0:00:10.753223950 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type sgpd
0:00:10.753299569 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type sbgp
0:00:10.753377492 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type nmhd
0:00:10.753454487 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux_types.c:239:qtdemux_type_get: unknown QuickTime node type tmcd
0:00:10.753944267 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux.c:3250:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:10.754570916 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux.c:3250:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
0:00:10.755072376 125225 0xaaaaf8e3c5e0 WARN                 qtdemux qtdemux.c:3250:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 3
Opening in BLOCKING MODE 
0:00:10.961130158 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:4561:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:10.961320949 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
0:00:10.961493404 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
0:00:10.961614785 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
0:00:10.961722341 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:11.109958696 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:4561:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:11.110108718 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
0:00:11.110243220 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
0:00:11.110416378 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
0:00:11.110532255 125225 0xfffef8052700 WARN                    v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xfffef001bc80 Failed to determine interlace mode
** INFO: <bus_callback:225>: Pipeline running

0:00:11.121794788 125225 0xfffef8052700 WARN            v4l2videodec gstv4l2videodec.c:2305:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
0:00:11.123285471 125225 0xfffef8052700 WARN          v4l2bufferpool gstv4l2bufferpool.c:1114:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:11.186672210 125225 0xfffeec00b2a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1565:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
**PERF:  19.10 (18.92)	
**PERF:  19.26 (19.12)	
**PERF:  19.26 (19.15)	
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:00:27.290968890 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:2435:gst_nvinfer_output_loop:<primary_gie> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:00:27.291547249 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1884> [UID = 1]: Tried to release an outputBatchID which is already with the context
ERROR from primary_gie: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(2435): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:00:27.294359138 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:2435:gst_nvinfer_output_loop:<primary_gie> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
ERROR from primary_gie: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(2435): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
0:00:27.294862102 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1884> [UID = 1]: Tried to release an outputBatchID which is already with the context
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:00:27.297528833 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:2435:gst_nvinfer_output_loop:<primary_gie> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:00:27.297833197 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1884> [UID = 1]: Tried to release an outputBatchID which is already with the context
ERROR from primary_gie: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(2435): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR: Failed to make stream wait on event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
0:00:27.300328978 125225 0xaaaaf8e3c120 WARN                 nvinfer gstnvinfer.cpp:1404:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
Quitting
ERROR: Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:00:27.300882408 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:2435:gst_nvinfer_output_loop:<primary_gie> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:00:27.301202581 125225 0xaaaaf8e3c240 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1884> [UID = 1]: Tried to release an outputBatchID which is already with the context
ERROR from primary_gie: Failed to queue input batch for inferencing
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1404): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR from primary_gie: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(2435): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR: Failed to make stream wait on event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
0:00:27.632871200 125225 0xaaaaf8e3c120 WARN                 nvinfer gstnvinfer.cpp:1404:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
ERROR: Failed to make stream wait on event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
0:00:27.633232239 125225 0xaaaaf8e3c120 WARN                 nvinfer gstnvinfer.cpp:1404:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
nvstreammux: Successfully handled EOS for source_id=0
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
0:00:27.828095650 125225 0xfffeec00a8c0 WARN            audiodecoder gstaudiodecoder.c:1753:gst_audio_decoder_drain:<faad0> still 1 frames left after draining
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::42] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)

It looks like we can also trigger this error when building a TRT engine. I have never seen this error occur on a 32GB AGX, but seems to occur ~30% of the time on the 64GB encrypted AGX.

[I] Building engine with configuration:
    Flags                  | [FP16]
    Engine Capability      | EngineCapability.DEFAULT
    Memory Pools           | [WORKSPACE: 63219.28 MiB]
    Tactic Sources         | [CUBLAS, CUBLAS_LT, CUDNN, EDGE_MASK_CONVOLUTIONS, JIT_CONVOLUTIONS]
    Profiling Verbosity    | ProfilingVerbosity.DETAILED
[E] 1: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 0x00000000000003e8 due to exception an illegal memory access was encountered
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Requested amount of GPU memory (534773760 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[W] Skipping tactic 1 due to insufficient memory on requested size of 534773760 detected for tactic 0x00000000000003ea.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 2 due to insufficient memory on requested size of 534773760 detected for tactic 0x0000000000000000.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Requested amount of GPU memory (100270080 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[W] Skipping tactic 0 due to insufficient memory on requested size of 100270080 detected for tactic 0x00000000000003e8.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 1 due to insufficient memory on requested size of 100270080 detected for tactic 0x00000000000003ea.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 2 due to insufficient memory on requested size of 100270080 detected for tactic 0x0000000000000000.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Requested amount of GPU memory (300810240 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[W] Skipping tactic 0 due to insufficient memory on requested size of 300810240 detected for tactic 0x00000000000003e8.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 1 due to insufficient memory on requested size of 300810240 detected for tactic 0x00000000000003ea.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 2 due to insufficient memory on requested size of 300810240 detected for tactic 0x0000000000000000.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Requested amount of GPU memory (66846720 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[W] Skipping tactic 0 due to insufficient memory on requested size of 66846720 detected for tactic 0x00000000000003e8.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 1 due to insufficient memory on requested size of 66846720 detected for tactic 0x00000000000003ea.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[W] Skipping tactic 2 due to insufficient memory on requested size of 66846720 detected for tactic 0x0000000000000000.
    Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[E] 1: [resizingAllocator.cpp::allocate::62] Error Code 1: Cuda Runtime (an illegal memory access was encountered)

We just saw the same error occur on an unencrypted 64GB AGX. So it looks like the encryption does not matter, seems to be purely whether it is 32GB or 64GB

  1. Do you have multiple AGX Xaviers? Do they flash the same fireware and Jetpack versions?

  2. Can you try upgrading to DS-6.4? There are no other users reporting similar issues on the forum.This may be caused by the GPU driver.

In addition, are you using the board and BSP provided by a third party?

We have multiple AGX Xaviers, flashing the same firmware and Jetpack versions. We are using a board and BSP provided by a third party. Recently we have been attempting to replicate on an AGX dev kit and are not seeing the error yet. This seems to indicate the problem may be due to the third party

OK. If you can reproduce the problem with AGX dev kit, you can reply to the topic or open a new topic.thx

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.