Video scaling with GStreamer element 'videoscale' and 'nvdrmvideosink set-mode=1'

Hi,

I am trying to up-scale the video input (2504x1438) to match the native display resolution (3840x2160) while maintaining the aspect-ratio. For this purpose, I use the following GStreamer pipeline:

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA' ! videoscale add-borders=true ! 'video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160' ! nvdrmvideosink set-mode=1 color-range=1

This pipeline generates the following output:

Setting pipeline to PAUSED ...
Using VT number 1
/dev/tty1 is in graphics mode
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvV4l2CameraSrc:nvv4l2camerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)2504, height=(int)1438, format=(string)BGRx, interlace-mode=(string)progressive, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)2504, height=(int)1438, format=(string)BGRx, interlace-mode=(string)progressive, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)2504, height=(int)1438, format=(string)BGRx, interlace-mode=(string)progressive, framerate=(fraction)30/1
ERROR: from element /GstPipeline:pipeline0/GstNvV4l2CameraSrc:nvv4l2camerasrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvV4l2CameraSrc:nvv4l2camerasrc0:
streaming stopped, reason not-negotiated (-4)
EOS on shutdown enabled -- waiting for EOS after Error
Waiting for EOS...

It turns out every scaling attempt fails. Only when keeping the input resolution, the pipeline executes successfully. I also used nvvidconv for scaling according to NVIDIA Jetson Linux Driver Package Software Features with nvdrmvideosink but ended up with the same error.

Therefore, I simplified the pipeline to research the error and realized that scaling doesn’t work when setting the option set-mode=1. But we need to set the color range and therefore this option has to be set.

So, the questions are:

  1. Is there a way to use videoscale with nvdrmvideosink set-mode=1 color-range=1?
  2. Is there another way to change the color range (X11 Window System deactivated)?

Best regards
Bruno

Hi,

The nvvidconv plugin supports scaling. Please try

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA' ! nvdrmvideosink set-mode=1 color-range=1

If it works, please then try

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)3840, height=(int)2160' ! nvdrmvideosink set-mode=1 color-range=1

Not sure about this question. For running nvdrmvideosink, you would need to disable X11 first.

Hi @DaneLLL,

the first pipeline

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA' ! nvdrmvideosink set-mode=1 color-range=1

works fine!
Running the second pipeline

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)3840, height=(int)2160' ! nvdrmvideosink set-mode=1 color-range=1

throws me the error from above:


To clarify the last question:

It seems that scaling is not working with nvdrmvideosink when using the option to set color range. As shown above I receive an error as soon as I scale the video.
Without changing the color range (nvdrmvideosink without option set-mode=1)

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)3840, height=(int)2160' ! nvdrmvideosink

the pipeline executes without error. Still not the desired behavior of the video output but the pipeline works.

Therefore, using a different way to change the color range would solve the problem. Is there a way to set the color range from within the driver?

Best regards
Bruno

Hi,
Please try with fakesink:

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)3840, height=(int)2160' ! fakesink

Not sure but BGRx may be a wrong format since default nvv4l2camerasrc only supports UYVY.

For launching v4l2 source, please refer to steps in Jetson Nano FAQ

Hi @DaneLLL,

I don’t know why I should try fakesink as the pipeline works with nvdrmvideosink as long as I don’t set the color range.

But I ran the pipeline with fakesink and the pipeline works.

My problem is that the same pipeline doesn’t work when setting the color range with nvdrmvideosink set-mode=1 color-range=1.

BGRx also works as I modified the source code of the nvv4l2camerasrc accordingly.

This pipeline works:

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)3840, height=(int)2160' ! nvdrmvideosink

But when setting the color range the pipeline does not work:

sudo gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)BGRx, width=(int)2504, height=(int)1438, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)3840, height=(int)2160' ! nvdrmvideosink set-mode=1 color-range=1

It throws the following error:

VT_WAITACTIVE failed, err=Input/output error
Failed to acquire vt
KDSETMODE KD_TEXT failed, err=Bad file descriptor
ERROR: Pipeline doesn't want to pause.
ERROR: from element /GstPipeline:pipeline0/GstNvDrmVideoSink:nvdrmvideosink0: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.
Additional debug info:
gstbasesink.c(5265): gst_base_sink_change_state (): /GstPipeline:pipeline0/GstNvDrmVideoSink:nvdrmvideosink0:
Failed to start

Also, the options offset-x and offset-y aren’t working when set-mode=1

And this is not bound to the nvv4l2camerasrc as I face the same problems when using videotestsrc

@DaneLLL do you face the same issues with nvdrmvideosink on your Jetson TX2?

Hi,
Please try these:

$ sudo gst-launch-1.0  videotestsrc ! 'video/x-raw, width=800,height=600, framerates=60/1' ! nvvidconv ! nvdrmvideosink set-mode=1 color_range=1
$ sudo gst-launch-1.0  videotestsrc ! 'video/x-raw, width=800,height=600, framerates=60/1' ! nvvidconv ! nvdrmvideosink set-mode=1 color_range=0

We can run the command on TX2/r32.6.1.

Hi,

both of your pipelines work.

Can you try the following pipeline on your TX2 R32.6.1?

sudo gst-launch-1.0  videotestsrc ! 'video/x-raw, width=800,height=600, framerates=60/1' ! nvvidconv ! 'video/x-raw, width=800, height=600' ! nvdrmvideosink set-mode=1 color_range=1

I receive the following error on my TX2 R32.6.1:


And there is actually no scaling. The width and height of the nvvidconv caps are identical with the width and height of the source.

Are you experiencing the same issue on your system?

Best regards
Bruno

Hi,
The pipeline looks invalid. Please send NVMM buffer to nvdrmvideosink

Hi @DaneLLL,

sorry, I was moving too fast. My bad! But the actual error remains. I corrected the pipeline:

sudo gst-launch-1.0  videotestsrc ! 'video/x-raw, width=800,height=600, framerates=60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=800, height=600' ! nvdrmvideosink set-mode=1 color_range=1

Error mesages:


Are you having the same issues on your system?

Best regards
Bruno

For the sake of completeness…

The following proposition is not true:

At least not on my TX2 with L4T R32.6.1. nvdrmvideosink runs smoothly with X11 activated (sudo systemctl set-default graphical.target).

Hi,
We are able to run this:

$ sudo gst-launch-1.0  videotestsrc ! 'video/x-raw, width=800,height=600, framerates=60/1' ! nvvidconv ! 'video/xraw(memory:NVMM),width=800,height=600' ! nvdrmvideosink set-mode=1 color_range=1

We don’t expect DRM to be run along with another GUI system. There is guidance of disabling default X11 in document before using DRM in gstreamer or jetson_multimedia_api. It may not work properly if you still keep X11 enabled. Suggest you disable it for a try.

Hi @DaneLLL,

this pipeline works on my system too (don’t know what mistake I made before). But it only works when no scaling is applied. In cases I need to scale, like the following pipeline does, I get the errors mentioned!

sudo gst-launch-1.0 -e -v videotestsrc ! 'video/x-raw, width=800, height=600, framerates=60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=2504, height=1438' ! nvdrmvideosink set-mode=1 color-range=1

Is this pipeline working on your system?

Also, the offset (offset-x & offset-y options) is not applied when used together with set-mode=1 color-range=1. No errors are thrown but the offset is just not applied on the output frames.

I have X11 window system deactivated the whole time. Just tried to run the pipeline on another system with X11 activated.
Side note: X11 is automatically deactivated after starting a pipeline with nvdrmvideosink. After reboot X11 is active again.

Best regards
Bruno

Hi,
In our set up, the TV supports up to 1920x1080 and we are not able to run in 2504x1438. We are able to run in 1920x1080:

sudo gst-launch-1.0 -e -v videotestsrc ! 'video/x-raw, width=800, height=600, framerates=60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1080' ! nvdrmvideosink set-mode=1 color-range=1

Setting offset-x/offset-y along with set-mode=1 color-range=1 may not work on current release. If you have multiple video planes for rendering, the solution is to composite the video planes into single plane in nvcompositor and then render the single video plane.

1 Like

Hi @DaneLLL,

because we have quite complex setups I still have to search for a complete solution. Your last post helped clarify some essential aspects of the issue.

Thanks!

Best regards
Bruno

I realized that the documentation (gst-inspect-1.0 nvdrmvideosink) states that my issue is not solvable with nvdrmvideosink.


I misinterpreted the description but now it is clear that the resolution input to nvdrmvideosink has to be the same as the display resolution.

Recap of issue: up-scaling while maintaining the aspect ratio + video output with nvdrmvideosink.
Example: Input resolution is 800x600 and display resolution is 1920x1080. Input resolution has a different aspect ratio than display resolution. Therefore, the input is embedded into the output. In other words, black borders are added when displaying the video. For this example, to keep aspect ratio the scaled input has a resolution of 1440x1080.

As described above the input resolution of nvdrmvideosink needs to be 1920x1080 because this is the display resolution. Otherwise, an error is thrown. Accordingly, an error is thrown when keeping the aspect ratio because the video needs to be scaled to 1440x1080.

Maybe I can work around this problem with nvcompositor to create a single video plane (with a resolution matching the display resolution) out of two video planes:

  1. black plane with display resolution (e.g. 1920x1080)
  2. video plane with arbitrary resolution (e.g. 1440x1080).

@DaneLLL would you agree with that or am I missing something?

Best regards
Bruno

Hi,
Your comment looks correct. Since the resolution cannot be changed after pipeline is running, you would need to scale to 1440x1080 for the input.

It is more flexible in using jetson_multimedia_api. Can call NvBufferComposite() directly and change the resolution to fit aspect ratio frame by frame. However more coding is required since there is no existing plugins you can use directly. FYI, the nvv4l2camerasrc plugin is 12_camera_v4l2_cuda in jetson_multimedia_api.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.