TX1 hang issue

Hi Sirs,

I meet a hang issue when I capture from camera and stream it out via gst-rtsp-server for three or four days. I can see the HDMI display is frozen, the usb mouse and usb keyboard can’t get response. My TX1 environment is L4T 23.1.

The following code is what I test.

#include <gst/gst.h>
#include <gst/rtsp-server/rtsp-server.h>

#define DEFAULT_RTSP_PORT "8554"           
#undef WITH_AUTH

static char *port = (char *)DEFAULT_RTSP_PORT;

int main(int argc, char **argv)
{
    GMainLoop *loop;
    GstRTSPServer *server;
    GstRTSPMountPoints *mounts;
    GstRTSPMediaFactory *factory;
    GOptionContext *optctx;
    GError *error = NULL;

    optctx = g_option_context_new("RTSP Server\n");
    g_option_context_add_group(optctx, gst_init_get_option_group( ));
    if(!g_option_context_parse(optctx, &argc, &argv, &error)) {
        g_printerr("Error parsing options: %s\n", error->message);
        return -1;
    }
    g_option_context_free(optctx);

    loop = g_main_loop_new(NULL, FALSE);

    server = gst_rtsp_server_new( );
    g_object_set(server, "service", port, NULL);

    mounts = gst_rtsp_server_get_mount_points(server);

    factory = gst_rtsp_media_factory_new( );

    gst_rtsp_media_factory_set_launch(factory, "(  nvcamerasrc fpsRange=30.0 30.0  intent=3 ! nvvidconv flip-method=6 ! video/x-raw\(memory:NVMM\), width=640, height=480, format=I420, framerate=30/1 ! omxh264enc bitrate=8000000 ! video/x-h264, stream-format=byte-stream ! h264parse ! rtph264pay name=pay0 pt=96 )");
    gst_rtsp_mount_points_add_factory(mounts, "/test", factory);

    g_object_unref(mounts);

    gst_rtsp_server_attach(server, NULL);

    g_main_loop_run(loop);

    g_main_loop_unref(loop);

    return 0;
}

Hi poplar,
A few notes regarding the issue you encountered,

  • could you check your system memory and see if there is memory leak somewhere? (your application, NV user library or kernel driver side)

you could use any program to examine memory usage. tegrastats can also give you system RAM usage.

  • I know it takes time to reproduce the issue but is the symptom repeatable consistently or you saw this only one time? Had you ran it, say for 7 days, and it’s fine. Or running 3 or 4 days, same issue occurs? We did perform stress test, including long-duration test for our BSP but will check with our SQA team again.

  • We have a latest r24.1 release so hope you could migrate it to this version and see if the issue remains. There are lots of changes from r23.1 (then r23.2) to r24.1 for a lot of components. R24.1 can be downloaded from,
    https://developer.nvidia.com/embedded/linux-tegra

Hi chijen,

-for the memory usage on tx1, I think we don’t meet a memory leak issue.

the tegrastats result as following,
ubuntu@tegra-ubuntu:~$ ./tegrastats
RAM 1954/3853MB (lfb 29x4MB) cpu [0%,0%,0%,0%]@518 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [42%,11%,30%,18%]@403 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [33%,24%,29%,34%]@307 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [32%,24%,32%,24%]@307 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [31%,43%,27%,16%]@403 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [35%,31%,12%,28%]@403 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [25%,19%,34%,29%]@403 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [32%,22%,22%,15%]@614 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [50%,10%,14%,17%]@614 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [37%,9%,12%,26%]@307 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [26%,8%,38%,24%]@518 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [24%,21%,25%,38%]@204 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [20%,29%,39%,14%]@614 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [24%,12%,35%,20%]@307 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [27%,24%,12%,39%]@403 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [39%,16%,13%,26%]@403 GR3D 0%@76 EDP limit 0
RAM 1954/3853MB (lfb 29x4MB) cpu [32%,23%,29%,16%]@307 GR3D 0%@76 EDP limit 0

  • tx1 got this hang issue for 4 times, each time can’t run over 7 days.

  • I’ve updated TX1 to r24.1, and I will try to run the same application to check if it hang or not.