GetcudaImage error:

So just to enable you asap: What you can do like I described in 2) is just get the frame and just reject it if it’s too old.

You need to define ‘too old’ for yourself.

Since this is a synchronous system where camera reads independently at 30 FPS, the application on the other side grabs frames at a different rate, either you process fast enough to be in par with the camera or you need to reject frames.

This way, I think, enables you fastest and is also the easiest one to implement on your side.

That should help and I hope it works for you.

Fabian

How should I reject the old frames, just return like dwSensorCamera_returnFrame()?
And I can not judge which frame is old in buffer.

Regarding your last question for copying the dwCameraFrameHandle_t, I cannot think of one case where you would want to do that?

So let me think: If you want to copy your image to do something besides like modifying it, why don’t you get your image via getImage() and then call dwImage_copyConvert() afterwards?

For copying the camera frame handle itself you can just pass the pointer around. I have somehow the feeling you want to use multi threading here, so for this I already advised you to use Image Streamers.

Sorry that I cannot give you more information on this, more light from your side might help here…

Fabian

Just returning on dwSensorCamera_returnFrame() is one of many ways to approach it. For the time being you should go with this one.

Since you are able to timestamp the images and your processing time, it is trivial to put that into a control statement.

Fabian

Hi Fabian, let me clarify clear
I want to use multi threading, but some questions. I used producer-consumer mode to make thread safe.

//Thread 1:
while(true)
{
    dwCameraFrameHandle_t frame_handle;
    dwSensorCamera_readFrame(frame_handle, ..);
    dwImageHandle_t image_handle;
    dwSensorCamera_getImage(image_handle, DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, frame_handle);
    dwImage_copyConvert(frames_produced.frame_handle, image_handle, sdk);
}
//Thread 2:
while(true)
{
    dwImageCUDA *rgba_image;
    dwImage_getCUDA(rgba_image, frames_consumed.frame_handle);
    Process(rgba_image);//cost about 100ms
}

If run the procedure, it will crash. If do not run Process(), it runs well. I think in Process() we do some GPU operations, some GPU resources are occupied. Every 33ms do dwSensorCamera_getImage(image_handle, DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, frame_handle), since GPU resources occupied, the gpu operation dwSensorCamera_getImage fail.
Since we do dwSensorCamera_getImage in thread 1, we get dwImageHandle_t in thread 2 and then use dwImage_getCUDA(image, dwImageHandle_t) get cuda-image. So I want to do dwSensorCamera_getImage in thread 2, make all gpu operations in thread 2, so I should get dwCameraFrameHandle_t from thread 1.

And as before get image used cpu type dwSensorCamera_getImage(image_handle, DW_CAMERA_OUTPUT_NATIVE_PROCESSED, frame_handle);

//Thread 1:
while(true)
{
    dwCameraFrameHandle_t frame_handle;
    dwSensorCamera_readFrame(frame_handle, ..);
    dwImageHandle_t image_handle;
    dwSensorCamera_getImage(image_handle, DW_CAMERA_OUTPUT_NATIVE_PROCESSED, frame_handle);
    dwImage_copyConvert(frames_produced.frame_handle, image_handle, sdk);
}
//Thread 2:
while(true)
{
    dwImageStreamer_producerSend(frames_consumed.frame_handle, streamer_cuda);
    dwImageHandle_t frame_cuda;
    dwImageStreamer_consumerReceive(&frame_cuda, 66000, streamer_cuda);
    dwImageCUDA *rgba_image;
    dwImage_getCUDA(rgba_image, frame_cuda);
    Process(rgba_image);//cost about 100ms
    dwImageStreamer_consumerReturn(&frame_cuda, streamer_cuda);
    dwImageStreamer_producerReturn(nullptr, 66000, streamer_cuda);
}

Everything runs well, since we get cpu image used dwSensorCamera_getImage(image_handle, DW_CAMERA_OUTPUT_NATIVE_PROCESSED, frame_handle), no gpu operations.
Once change to get gpu image, run fail.

Wait, you are using the streamer on one thread only.

For multi-threading you need to use the following:

// Thread 1:
dwImageStreamer_initialize(...);
while(true)
{
    // CODE: Get an image to stream
    dwImageStreamer_producerSend(...);
    dwImageStreamer_producerReturn(TIMEOUT);
}
dwImageStreamer_release(...);
// Thread 2:
// CODE: If the streamer has been initialized on thread 1
while(true)
{
    dwImageStreamer_consumerReceive(TIMEOUT);
    // CODE: Use the streamed image
    dwImageStreamer_consumerReturn(...);
}

Please provide us also more information about errors after adapting to the above code.

Fabian

The error messages as the first floor “Driveworks exception thrown: DW_CUDA_ERROR: Call failed cuEGLStreamConsumerAcquireFrame : unspecified launch failure”.

I think dwImageStreamer_producerSend(frame_data.frame_handle, streamer_cuda)) is for convert from cpu to gpu. I use dwSensorCamera_getImage(image_handle, DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, frame_handle) getting gpu image, do not need to use dwImageStreamer_producerSend.

No, the dwImageStreamer is made for multi-threading. You are not thread safe on post #25 and hence your application will crash.

Converting from CPU to GPU is one of many purposes which you are not limited to exhaust. You need to adapt against the producer/consumer pattern, else your approach won’t work.

For this I refer you once again to our Image Streamer Multi and Cross Application samples.

Just modify the samples and it will work out of the box.

Fabian

ok, thanks.
So is the posted #26 thread safe?
So whatever I use dwSensorCamera_getImage with cpu or gpu type, I must use dwImageStreamer to ensure thread safe?

Yes you need to and #26 is exactly not thread safe but #27 then is.

Fabian

If I use the method you posted on #27, I still have problem, my consumer speed is much slower than producer speed, then the producer may be blocked, so may also ICP drops.
I think my posted #25 is thread safe, I already have used producer-consumer-pattern to ensure thread-safe, since do not run Process() everything runs well.

Using the interface dwSensorCamera_readFrame(), the images are put into buffer, a queue, then dwSensorCamera_getImage() get image from the queue? So is there a interface that we can get image in real-time, not the buffer? That is decrease the buffer size to 1, so it only contains recent images.

Hi,

#25 is not thread-safe.

The fact you are using a producer-consumer-pattern does not mean the producer nor the consumer is running for the same duration or even number of operations.

Your consumer is slower w.r.t. your producer, so just drop the frames and accept this cannot be avoided. You produce with 30Hz but consume with 100ms, so this is to be expected.

Fabian

So is there any reasonable solutions to solve this problem?

You can either optimize your OnProcess() method or accept the fact of ICP drops.

Fabian

OK, well.
Thanks!

As the codes posted on #25, as before it runs failed.
If I $export CUDA_VISIBLE_DEVICES=1 it run success but fps is 4.
if $export CUDA_VISIBLE_DEVICES=0 it run failed.
How did it happen?
Thanks!

Could you please provide the whole error log regarding your failure?

Else I cannot fully understand what’s going on on your side.

Fabian

The error likes below

cudnnEngine.cpp (269) - Cuda Error in enqueue: 4
cudnnEngine.cpp (269) - Cuda Error in enqueue: 4
CPU time             TensorRT: 116.820999 ms
Input velocity is invalid!
[26-6-2019 10:14:7] Driveworks exception thrown: DW_CUDA_ERROR: Call failed cuEGLStreamConsumerAcquireFrame : unspecified launch failure

terminate called after throwing an instance of 'std::runtime_error'
  what():  DW Error DW_CUDA_ERROR executing DW function:
 dwSensorCamera_getImage( &image_handle, DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, frame_handle)