DeepStream 8.0 (Docker on WSL2) – nvds_obj_enc_process returns error code 1 when saving images (Segmentation fault)

Hardware Platform: NVIDIA GPU (GeForce RTX 5080, running on PC with WSL2 + Docker)

Environment:

  • DeepStream Version: 8.0.0

  • JetPack Version: N/A (PC/WSL2 environment)

  • TensorRT Version: 10.9

  • CUDA Driver Version: 577.00

  • CUDA Runtime Version: 12.8

  • cuDNN Version: 9.8

  • libNVWarp360 Version: 2.0.1d3

  • Installation: DeepStream 8.0 Docker on WSL2

Issue Type: Bug


Description of the Issue:
I am running DeepStream 8.0 in Docker on WSL2 and trying to save cropped object images using nvds_obj_enc_process.

The encoder context is created successfully at startup in create_pipeline this function in deepstream_app.c:

appCtx->obj_ctx_handle = nvds_obj_enc_create_context(0);
if (!appCtx->obj_ctx_handle) {
    g_print("ERROR: Unable to create obj encoder context for GPU %d\n", gpu_id);
    return -1;
}
g_print("Successfully created obj encoder context\n");

Output confirms context creation:

Successfully created obj encoder context

But later, when calling nvds_obj_enc_process, the function fails:

DEBUG: Using surface index: 1
ERROR: nvds_obj_enc_process failed with return code: 1
 
  - Object ID: 3
  - Source ID: 1
Segmentation fault (core dumped)

Steps to Reproduce:

  1. Run DeepStream 8.0 Docker inside WSL2 with GPU passthrough.

  2. Add custom code to save cropped images with nvds_obj_enc_process.

  3. Run the custom app:

./deepstream-app-custom -c config/deepstream_app_config.txt

Observed Behavior:

  • nvds_obj_enc_process fails with return code: 1 even though frame_meta, obj_meta, and ip_surf are valid.

  • Application crashes with segmentation fault.


Expected Behavior:

  • The cropped object image should be saved successfully.

  • No segmentation fault should occur when encoder context has already been created.


Additional Notes:

  • I verified that frame_meta, obj_meta, and ip_surf are valid when passed to nvds_obj_enc_process.

  • Encoder context is created and confirmed, but still fails during image saving.

  • This issue only happens in Docker on WSL2 — it may be related to GPU driver passthrough or encoding libraries in WSL.
    this is the code:

    
    static void save_alert_image(AppCtx *appCtx, long int count, NvBufSurface *ip_surf,
                                NvDsFrameMeta *frame_meta, NvDsObjectMeta *obj_meta, bool calc_enc) {
                                    
        std::string temp_osd_string = "/opt/nvidia/deepstream/deepstream-8.0/images/"; 
        const char* osd_string = temp_osd_string.c_str();
        
        // Initialize frame data structure
        NvDsObjEncUsrArgs frameData;
        memset(&frameData, 0, sizeof(NvDsObjEncUsrArgs));
        
        frameData.isFrame = 0;  // 0 for object crop
        frameData.saveImg = true;
        frameData.attachUsrMeta = false;
        frameData.scaleImg = FALSE;
        frameData.scaledWidth = 0;
        frameData.scaledHeight = 0;
        frameData.quality = 80;
        
        if (calc_enc) {
            frameData.calcEncodeTime = 1;
        }
    
        snprintf(frameData.fileNameImg, sizeof(frameData.fileNameImg), "%s%ld.jpg", osd_string, count);
        guint surface_index = frame_meta->batch_id;
        if (surface_index >= ip_surf->numFilled) {
            g_print("Error: Invalid surface index %d (numFilled: %d)\n", 
                    surface_index, ip_surf->numFilled);
            return;
        }
        
        g_print("DEBUG: Using surface index: %d\n", surface_index);
        
        // Call encoder
        NvDsObjEncCtxHandle ctx = appCtx->obj_ctx_handle;
        gint ret = nvds_obj_enc_process(ctx, &frameData, ip_surf, obj_meta, frame_meta);
        
        if (ret != 0) {
            g_print("ERROR: nvds_obj_enc_process failed with return code: %d\n", ret);
            g_print("  - This usually means encoder context is not properly initialized\n");
            g_print("  - Object ID: %ld\n", obj_meta->object_id);
            g_print("  - Source ID: %d\n", frame_meta->source_id);
            g_print("  - Check encoder initialization in your main setup code\n");
            
            // DO NOT call finish after failure - this causes the segfault
            return;
        }
        
        g_print("DEBUG: nvds_obj_enc_process succeeded\n");
        
        // Only call finish if process succeeded
        nvds_obj_enc_finish(ctx);
        
        g_print("SUCCESS: Alert image saved at %s%ld.jpg\n", osd_string, count);
        std::cout << "image_saved---------------------------" << std::endl;
    }
    
    extern "C" void
    
    parse_nvdsanalytics_meta_data_deepstream(GstBuffer *buf,AppCtx *appCtx,NvDsBatchMeta *batch_meta)
    {
        const gchar *calc_enc_str = g_getenv("CALCULATE_ENCODE_TIME");
        gboolean calc_enc = !g_strcmp0(calc_enc_str, "yes");
    
        GstMapInfo inmap = GST_MAP_INFO_INIT;
          if (!gst_buffer_map(buf, &inmap, GST_MAP_READ))
            {
                GST_ERROR("input buffer mapinfo failed");
            }
        NvBufSurface* ip_surf = (NvBufSurface *)inmap.data;
    
       if (appCtx->obj_ctx_handle == NULL) {
            g_print("Error: obj_ctx_handle is NULL. Did you initialize it?\n");
            gst_buffer_unmap(buf, &inmap);
            return;
        }
        
        NvDsObjectMeta *obj_meta = NULL;
        NvDsMetaList * l_frame = NULL;
        NvDsMetaList * l_obj = NULL;
        NvDsMetaList *l_user = NULL;
        // auto current_time = std::chrono::steady_clock::now();
        // auto time_elapsed = std::chrono::duration_cast<std::chrono::seconds>(current_time-last_sent_time);
        //NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
    
        for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
                l_frame = l_frame->next) {
            NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
            int camera_id = frame_meta->source_id;
            char fileObjNameString[256];
            int num_rect = 0;
            for (NvDsMetaList *l_obj = frame_meta->obj_meta_list;l_obj!=NULL;l_obj=l_obj->next){
                NvDsObjectMeta *obj_meta = (NvDsObjectMeta *)(l_obj->data);
                int gie_unique_id = obj_meta->unique_component_id;
                if (gie_unique_id==2){
                    std::cout << "SECONDARY Model - "
                          << "Class ID: " << obj_meta->class_id 
                          << ", Tracking ID: " << obj_meta->object_id 
                          << ", Confidence: " << obj_meta->confidence
                          << ", Parent ID: " << obj_meta->parent->object_id  // Link to primary detection
                          << std::endl; 
                        std::string unique_id = std::to_string(obj_meta->parent->object_id)+"_"+std::to_string(obj_meta->class_id);
                        std::cout << "Unique Secondary ID: " << unique_id << std::endl;
                        save_alert_image(appCtx,obj_meta->parent->object_id,ip_surf,frame_meta,obj_meta->parent,calc_enc);
    
                }
           
            }
        }
        gst_buffer_unmap(buf, &inmap);
    
    }
    

Can deepstream-image-meta-testwork normally?

This may be related to wsl2, I will try to reproduce it. If these problems block you, you can use native ubuntu linux for development

Thanks for the suggestion! I tested using the deepstream-image-meta-test sample under WSL2 Docker (DeepStream 8.0) as you recommended.

When I run:

/opt/nvidia/deepstream/deepstream-8.0/sources/apps/sample_apps/deepstream-image-meta-test# \
./deepstream-image-meta-test 0 file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264

I get the following output:

max_fps_dur 8.33333e+06 min_fps_dur 2e+08
WARNING: Overriding infer-config batch-size (2) with number of sources (1)

(deepstream-image-meta-test:732): GLib-GObject-CRITICAL **: g_object_set_is_valid_property: object class 'GstFakeSink' has no property named 'gpu-id'
Now playing...
Got EOS from stream
ERROR: nvds_obj_enc_process failed with return code: 1
  - This usually means encoder context is not properly initialized
Segmentation fault (core dumped)

So it seems the same encoder initialization error and segmentation fault occur even in the official sample.

Here’s the relevant part of the sample code (deepstream-image-meta-test):

 gint ret = nvds_obj_enc_process (ctx, &frameData, ip_surf, NULL, frame_meta);
if (ret != 0) {
    g_print("ERROR: nvds_obj_enc_process failed with return codes: %d\n", ret);
    g_print("  - This usually means encoder context is not properly initialized\n");
}

...
gint ret = nvds_obj_enc_process (ctx, &objData, ip_surf, obj_meta, frame_meta);
if (ret != 0) {
    g_print("ERROR: nvds_obj_enc_process failed with return codeq1: %d\n", ret);
    g_print("  - This usually means encoder context is not properly initialized\n");
}

It runs fine until the EOS event, then fails exactly when the encoding function is called.
Everything (model loading, inference, and metadata) works correctly up to that point.

This is a legacy bug. If we fix it, I will post a response here.
DeepStream on WSL is in alpha release, so please use native ubuntu development first.

There are still some limitations in WSL, please refer to this link

  • Image encode not supported in WSL.
  • In WSL2, black screen with log “”MESA: error: Failed to attach to x11 shm” is observed with pipelines with nveglglessink or other display sinks. Use filesink instead in such usecases.