BUG：UserMetadata.set_user_data_json

[cihn]

[Bug][DeepStream 9.1][pyservicemaker] UserMetadata.set_user_data_json() causes a double free during metadata release

I encountered a reproducible process abort when using
UserMetadata.set_user_data_json() in pyservicemaker. I reduced the issue to
the following minimal pipeline and investigated the metadata release path.

Environment

• Hardware Platform (Jetson / GPU): GPU, x86_64, NVIDIA Tesla T4
• DeepStream Version: 9.1.0
• JetPack Version (valid for Jetson only): N/A
• TensorRT Version: 10.16
• NVIDIA GPU Driver Version: 590.48.01
• CUDA Version reported by nvidia-smi: 13.1
• CUDA Driver Version reported by deepstream-app in the container: 13.2
• CUDA Runtime Version: 13.2
• CUDA Forward Compatibility: Enabled
• cuDNN Version: 9.20
• Operating System: Ubuntu 24.04.4 LTS, x86_64
• Python Version: 3.12.3
• pyservicemaker Version: 0.0.1
• Docker Image: nvcr.io/nvidia/deepstream:9.1-triton-multiarch
• Docker Image Digest: sha256:fd31f5b44ababdbdee8cd397a375e888191b49e402ac237254a4cdc239130f5b
• Issue Type: Bug

The host nvidia-smi reports CUDA 13.1. The container enables CUDA Forward
Compatibility, so deepstream-app reports the loaded CUDA driver and runtime
as 13.2.

Summary

Calling UserMetadata.set_user_data_json() causes the process to abort with:

free(): double free detected in tcache 2

The JSON metadata is successfully attached and read by a downstream probe
before the crash. The same crash occurs when the downstream reader is disabled,
so get_user_data_json() and JSON parsing are not required to trigger it.

The reproducer uses one source, one generic UserMetadata, a linear pipeline,
and the sample MP4 shipped in the official DeepStream container. It does not use
inference, tracking, OSD, tee, a message converter, a message broker, or a
custom C++ plugin.

Expected behavior

The following lifecycle should complete normally and exit with status 0:

BatchMetadata.acquire_user_meta()
-> UserMetadata.set_user_data_json(payload, meta_type)
-> ObjectMetadata.append(user_meta)
-> FrameMetadata.append(object_meta)
-> downstream get_user_data_json()
-> normal metadata release
-> EOS

Actual behavior

The payload is written and read successfully, but metadata release aborts the
process with SIGABRT and shell exit status 134:

WRITE payload={'kind': 'user_metadata_reproducer', 'source_id': 0, 'frame_number': 0}
READ payload={'kind': 'user_metadata_reproducer', 'source_id': 0, 'frame_number': 0}
free(): double free detected in tcache 2
Aborted (core dumped)
exit_status=134

How to reproduce

1. Save the reproducer

Save the following code as
➡️ test_deepstream9_1_user_metadata_double_free.py:

#!/usr/bin/env python3
"""Minimal reproducer for pyservicemaker UserMetadata JSON double-free."""

import argparse
import os

from pyservicemaker import BatchMetadataOperator, Pipeline, Probe


USER_META_TYPE = 8193
DEFAULT_INPUT = (
    "/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4"
)


class UserMetadataWriter(BatchMetadataOperator):
    """Attach one generic JSON UserMetadata instance to an object."""

    def __init__(self):
        super().__init__()
        self.written = False

    def handle_metadata(self, batch_meta):
        if self.written:
            return

        for frame_meta in batch_meta.frame_items:
            payload = {
                "kind": "user_metadata_reproducer",
                "source_id": int(frame_meta.source_id),
                "frame_number": int(frame_meta.frame_number),
            }

            # FrameMetadata.append(UserMetadata) is rejected by the Python binding,
            # so use the supported ObjectMetadata.append(UserMetadata) overload.
            object_meta = batch_meta.acquire_object_meta()
            object_meta.class_id = 0
            object_meta.confidence = 1.0
            object_meta.rect_params.left = 0.0
            object_meta.rect_params.top = 0.0
            object_meta.rect_params.width = 1.0
            object_meta.rect_params.height = 1.0
            object_meta.label = "user-meta-carrier"

            user_meta = batch_meta.acquire_user_meta()
            user_meta.set_user_data_json(payload, USER_META_TYPE)
            object_meta.append(user_meta)
            frame_meta.append(object_meta)

            self.written = True
            print(f"WRITE payload={payload}", flush=True)
            return


class UserMetadataReader(BatchMetadataOperator):
    """Read the JSON payload from a downstream probe."""

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            for object_meta in frame_meta.object_items:
                for user_meta in object_meta.user_meta_items(USER_META_TYPE):
                    payload = user_meta.get_user_data_json()
                    print(f"READ payload={payload}", flush=True)


def to_uri(path):
    if path.startswith(("file://", "rtsp://", "http://", "https://")):
        return path
    return "file://" + os.path.abspath(path)


def run_pipeline(input_path, no_reader):
    pipeline = Pipeline("user-metadata-double-free-reproducer")
    pipeline.add("nvurisrcbin", "src", {"uri": to_uri(input_path)})
    pipeline.add(
        "nvstreammux",
        "mux",
        {
            "batch-size": 1,
            "width": 1280,
            "height": 720,
            "batched-push-timeout": 33000,
        },
    )
    pipeline.add("queue", "writer_queue")
    pipeline.add("queue", "reader_queue")
    pipeline.add("fakesink", "sink", {"sync": 0, "async": 0})

    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "writer_queue", "reader_queue", "sink")
    pipeline.attach(
        "writer_queue",
        Probe("user_metadata_writer", UserMetadataWriter()),
    )
    if not no_reader:
        pipeline.attach(
            "reader_queue",
            Probe("user_metadata_reader", UserMetadataReader()),
        )

    pipeline.start().wait()


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", default=DEFAULT_INPUT)
    parser.add_argument(
        "--no-reader",
        action="store_true",
        help="Remove the downstream reader; the metadata release still crashes.",
    )
    args = parser.parse_args()
    run_pipeline(args.input, args.no_reader)


if __name__ == "__main__":
    main()

The metadata type 8193 is NVDS_START_USER_META
(4096 + 4096 + 1) in DeepStream 9.1. The API allows application-specific
metadata types greater than or equal to this value.

2. Start the official DeepStream 9.1 container

Run this command from the directory containing the saved Python file:

docker run --rm -it \
  --gpus all \
  -v "$PWD:/workspace:ro" \
  -w /workspace \
  nvcr.io/nvidia/deepstream:9.1-triton-multiarch \
  bash

3. Install the required Python dependency

The tested container does not include the yaml Python module, which
pyservicemaker imports during initialization:

python3 -m pip install --no-cache-dir pyyaml

This dependency is required only to import pyservicemaker and is unrelated to
the metadata crash.

4. Confirm the environment and run

deepstream-app --version-all
python3 --version
python3 -c "import importlib.metadata; print(importlib.metadata.version('pyservicemaker'))"

python3 test_deepstream9_1_user_metadata_double_free.py
echo "exit_status=$?"

The script has no required arguments and uses the official sample MP4 by
default. The relevant output is:

deepstream-app version 9.1.0
DeepStreamSDK 9.1.0
CUDA Driver Version: 13.2
CUDA Runtime Version: 13.2
TensorRT Version: 10.16
cuDNN Version: 9.20
Python 3.12.3
0.0.1

WRITE payload={'kind': 'user_metadata_reproducer', 'source_id': 0, 'frame_number': 0}
READ payload={'kind': 'user_metadata_reproducer', 'source_id': 0, 'frame_number': 0}
free(): double free detected in tcache 2
Aborted (core dumped)
exit_status=134

The container may also warn that no decoder is available for the MP4’s AAC
audio track. The video pad is created successfully, and this warning is
unrelated to the crash.

5. Run the no-reader control

python3 test_deepstream9_1_user_metadata_double_free.py --no-reader
echo "exit_status=$?"

Result:

WRITE payload={'kind': 'user_metadata_reproducer', 'source_id': 0, 'frame_number': 0}
free(): double free detected in tcache 2
Aborted (core dumped)
exit_status=134

This confirms that the downstream reader and get_user_data_json() are not
required to trigger the crash.

Optional script arguments:

Argument	Purpose
--no-reader	Disable the downstream reader probe.
--input /path/to/video.mp4	Use another input path accessible inside the container.
--help	Show the built-in help.

No model, inference configuration, or external service is required.

Root-cause evidence

The basic reproduction steps above are sufficient to trigger the bug. The
following information is included only to help identify the faulty release
path.

GDB backtrace

gdb -q -batch \
  -ex 'set pagination off' \
  -ex run \
  -ex 'thread apply all bt' \
  --args python3 test_deepstream9_1_user_metadata_double_free.py --no-reader

Relevant stack:

Thread "reader_queue:sr" received signal SIGABRT, Aborted.

#0  pthread_kill
#1  raise
#2  abort
#6  free
#7  nvds_destroy_meta_pool
#8  nvds_destroy_batch_meta
#9  gst_buffer_foreach_meta
#10 gst_buffer_pool_release_buffer
#12 gst_mini_object_unref

reader_queue is the remaining GStreamer queue element; --no-reader removes
the Python reader probe. The second free is detected while DeepStream destroys
the batch metadata pool.

Release callback argument

A breakpoint on release_cstring_user_data(void*, void*) showed:

META_TYPE=8193
USER_DATA_PTR=<pointer to the intact JSON string>

The callback argument has the layout of a valid NvDsUserMeta, and its
user_meta_data field points to the JSON string. Therefore, the helper receives
NvDsUserMeta*, not the JSON char*.

The affected binary is:

/usr/local/lib/python3.12/dist-packages/pyservicemaker/_pydeepstream.so
size: 7257528 bytes
sha256: b40dae01817ee71fa22ff200896935fdc86bf49c1666f0b5d02d0eb52f61cacf

Its local helper symbols are:

00000000001288c2 t copy_cstring_user_data(void*, void*)
00000000001288ec t release_cstring_user_data(void*, void*)

Disassembly shows that the helpers pass their first callback argument directly
to strdup() and free() respectively.

Metadata callback contract

The official DeepStream 9.1 sample
deepstream-user-metadata-test/deepstream_user_metadata_app.c states that the
first callback argument is NvDsUserMeta* and dereferences
user_meta->user_meta_data:

static gpointer copy_user_meta(gpointer data, gpointer user_data)
{
  NvDsUserMeta *user_meta = (NvDsUserMeta *)data;
  gchar *src_user_metadata = (gchar *)user_meta->user_meta_data;
  /* Allocate and copy src_user_metadata. */
}

static void release_user_meta(gpointer data, gpointer user_data)
{
  NvDsUserMeta *user_meta = (NvDsUserMeta *)data;
  if (user_meta->user_meta_data) {
    g_free(user_meta->user_meta_data);
    user_meta->user_meta_data = NULL;
  }
}

The observed pyservicemaker helper behavior is equivalent to:

void* copy_cstring_user_data(void* data, void*) {
    return strdup(static_cast<char*>(data));
}

void release_cstring_user_data(void* data, void*) {
    free(data);
}

The release helper frees the NvDsUserMeta pool object itself instead of its
user_meta_data. DeepStream later destroys the same pool object, producing the
double free. The copy helper has the corresponding pointer-level problem when a
real metadata deep copy invokes it.

Regression information

The same reproducer also aborts with exit status 134 on DeepStream 9.0 and
pyservicemaker 0.0.1. The tested DeepStream 9.0 binary hash is:

sha256: 6d00a99f0070768dea763494dcfb5e8ddd543e5ae8defadc025b245f2aa9147b

Requested fix

Please update the JSON copy and release helpers to dereference
NvDsUserMeta::user_meta_data and use matching allocation and deallocation
functions. Conceptually:

void* copy_cstring_user_data(void* data, void*) {
    auto* user_meta = static_cast<NvDsUserMeta*>(data);
    const char* payload = static_cast<const char*>(user_meta->user_meta_data);
    return payload ? strdup(payload) : nullptr;
}

void release_cstring_user_data(void* data, void*) {
    auto* user_meta = static_cast<NvDsUserMeta*>(data);
    free(user_meta->user_meta_data);
    user_meta->user_meta_data = nullptr;
}

The allocator and deallocator should match those used by the binding’s actual
JSON string allocation path.

Impact

This bug prevents Python Service Maker applications from safely attaching JSON
user metadata for downstream processing. The payload appears usable at first,
but releasing the metadata causes a fatal process-level abort.

Official API references

• Service Maker for Python Developers: https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_service_maker_python.html
• deepstream::UserMetadata API: https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/classdeepstream_1_1UserMetadata.html
• DeepStream metadata structures and callbacks: https://docs.nvidia.com/metropolis/deepstream/dev-guide/sdk-api/group__metadata__structures.html

[Fiona.Chen]

@cihn

• DeepStream Version: 9.1.0
• JetPack Version (valid for Jetson only): N/A
• TensorRT Version: 10.16
• NVIDIA GPU Driver Version: 590.48.01

@cihn

The host nvidia-smi reports CUDA 13.1. The container enables CUDA Forward
Compatibility, so deepstream-app reports the loaded CUDA driver and runtime
as 13.2.

Please refer to the DeepStream 9.1 compatibilities.

Thank you for reporting this issue, I can reproduce it. It seems a UserMetadata.set_user_data_json binding implementation issue. We will fix it in the future.

Before the issue is fixed, the following workaround may be options:

• Carry your JSON out-of-band (e.g. a Python dict keyed by frame_number/source_id in your probe) instead of attaching it as DeepStream user metadata, or
• Use a purpose-built native user-meta type (EventMessageUserMetadata, tensor/segmentation metas, etc.) whose callbacks are implemented correctly, or
• If you must ship JSON downstream to C/nvmsgconv, use the NVDS_CUSTOM_MSG_BLOB path used by nvmsgconv rather than the JSON binding.