Wierdly the pipeline works for the first 718 frames and then exactly at the 719th frame throws this error
E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-07-10 10:22:44.807278: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1
Aborted (core dumped)
It seems to be something to do with cuda and tensorflow but I don’t get why the ssd-parser example was working then
Step 1 : **This pipeline code is almost exactly the same as the pipeline for the imagedata example **
**the only change I made here was to add the pgie probe from the deepstream ssd example and change inference element to infer server. ** The config I am using is exactly the same as the one in the ssd example
Step 2 : tiler`probe function code :
here the changes I made from the image data example are only in cases of accessing the class name of corresponding object ids and vice-versa as classes for both models are different
the error I get occurs on the exact same frame everytime
In an attempt to solve this problem what I tried was working from bottom to up.
I started off with the ssd-example you guys provided and iteratively added features from the image-data example.
This code succesfully processes all frames of the video but runs into to following error whenever I try to process image using opencv
this function call : n_frame=pyds.get_nvds_buf_surface(hash(gst_buffer),frame_meta.batch_id)
leads to a segmentation fault
Regarding your questions about where to put the file
I’m running the code on a docker container of a vm instance in google cloud.
Please download those two file I had shared
1: deepstream-ssd-image.py(deepstream-ssd-image.py - Google Drive )
2: deepstream-ssd-parser-multi-src.py(deepstream-ssd-parser-multi-src.py - Google Drive)
and place them in the folder
sources/python/apps/deepstream-ssd -parser (folder was generated for me when I extracted the python bindings )
also add sample_720p.h264 from the samples/streams folder to this folder
once in the folder
please run : python3 deepstream-ssd-image.py file:///opt/nvidia/deepstream/deepstream-5.0/sources/python/apps/deepstream-ssd-parser/sample_720p.h264 frames
to create issue 1(Illegal Access)
and run
python3 deepstream-ssd-parser-multi-src.py file:///opt/nvidia/deepstream/deepstream-5.0/sources/python/apps/deepstream-ssd-parser/sample_720p.h264 frames
Hey I think I know what is causing both issues
This piece of code
if not is_aarch64():
# Use CUDA unified memory in the pipeline so frames
# can be easily accessed on CPU in Python.
mem_type = int(pyds.NVBUF_MEM_CUDA_UNIFIED)
streammux.set_property(“nvbuf-memory-type”, mem_type)
nvvidconv.set_property(“nvbuf-memory-type”, mem_type)
nvvidconv1.set_property(“nvbuf-memory-type”, mem_type)
tiler.set_property(“nvbuf-memory-type”, mem_type)
if it is included leads to illegal access and if not causes the segmentation fault
I guess it has to be included as without this the frames cannot be read using the get_nvds_buf_surface function without this
But do you know why it may be causing the illegal access error?
Hi @shettyashwath2010,
Thanks for the repo steps!
I can reproduce the issue on dGPU platform. I’ll try on Jetson later, and then see how to debug this issue.
And, thanks for the TF link.
BTW, in this post, someone suggested to try new TF.
Hey I am working with dgpu platform only. So if you are able to find solution for that it’s perfect.
Thanks for the link ill try updating tensorflow, but something I’ve noticed is the ds-triton docker container does not contain the tf library in python ( it says module not found when I try to import), I see it is present in the form of a .so file Could you advice me how to update in such cases?
If I install the newer version of tf in python3 will the change be reflected on deep stream ( I doubt this will work)
please check the change in attached file and try again with below command for sample_720p.h264
/opt/nvidia/deepstream/deepstream-5.0/sources/python/apps/deepstream-ssd-parser# rm -rf frames; rm ~/.cache/gstreamer-1.0/*
/opt/nvidia/deepstream/deepstream-5.0/sources/python/apps/deepstream-ssd-parser# python3 deepstream-ssd-image.py file:///opt/nvidia/deepstream/deepstream-5.0/sources/python/apps/deepstream-ssd-parser/sample_720p.h264 frames
Hey, could I know what do you think was the technical reason behind the error?
I briefly looked at the changes you made, it seems to be primarily in the resolution of the streammux and tiler.
Is it something I would have to change according to the video source I am using?
Also if you don’t mind me asking could I know any resources you suggest looking into so that I can debug errors like these on my own ?
After further debug, I found that juts changing “streammux.set_property(‘height’, 1080)” to “streammux.set_property(‘height’, 1088)” or some other values, e.g. 1096, the crash is gone.
And, “streammux.set_property(‘height’, 1080)” works on Jetson platform, so seems there is issue in tf preprocessing on dGPU platfrom.
You can verify this WAR and use it temporarily.
We will check internall if this is a known issue.
Hi @shettyashwath2010
This can be also solved by removing below line, since tiler connects to nvvidconv, leave the mem-type to be negotiated by these two plugins.
Also I have some queries regarding efficiently communicating between processes using deepstream? Would you recommend creating another thread for this?. I am trying to send the entire frame and data accessed in the probe function to another parallelly running process (using python multiprocessing library) . I have tried using shared memory through multiprocessing queues and server process managers and all of them heavily bottleneck the fps. What would you suggest trying?