How does Jetson NVENC compare with Desktop NVENC?

I am considering porting my project (a Slingbox-style TV private distribution system) to Jetson and had assumed that since Jetson is using NVENC chips that I’d be able to use the same Video SDK I already use for the consumer GPU cards for HEVC, AV1 etc.

It seems not, and there is a different approach for Jetson.

Can anyone clarify whether the hardware is the same but the SDK different, or is Jetson using different encoder hardware altogether? More to the point, can I expect comparable quality encoding using Jetson as I would Desktop or has it be dumbed down because it’s aimed at different use cases?

Hi,
Hardware block is identical but clock is different. For capability of Orin NX, please refer to module data sheet:
https://developer.nvidia.com/downloads/jetson-orin-nx-module-series-data-sheet

Besides, software frameworks are different. We support gstreamer and jetson_multimedia_api on Jetson platforms.

Thank you for that.

I am looking to pair it with a Magewell M.2. capture device, and continue to use the Magewell SDK for capture as I have already written my code around that for the Desktop version so it would be convenient to continue using it.

Is there a prepared approach for taking these captured video frames (in system memory) and encoding using streamer or Jetson_multimedia_api, i.e. for this use case which would be best, of those two options?

Hi,
A possible solution is to integrate Magewell SDK with jetson_multimedia_api. Please refer to the samples in

/use/src/jetson_multimedia_api

For hardware encoding function, please check 01_video_encode sample.

1 Like

Thank you, will check that out.

I cannot see equivalents in the Multimedia API to the following that I use in the VIDEO SDK:

  • NV_ENC_TUNING_INFO (low latency vs high quality,. etc)
  • NV_ENC_CONFIG.rcParams.enableLookahead and lookaheadDepth
  • hevcVUIParameters.colourDescriptionPresentFlag + related fields that enable writing colour primaries, colour matrix, transfer characteristics etc into the bitstream
  • equivalent to above for AV1
  • nvEncGetSequenceParams() - this allows retrieving the SPSPPS data after encoder initialisation.

Are these not supported, and if not are there plans to support them?

Also, there is no function inside NvVideoEncoder that controls this flag:

V4L2_CID_MPEG_VIDEOENC_AV1_HEADERS_WITH_FRAME

It looks like it is missing something like

int setAv1HeadersWithFrameEnabled(bool enabled);

Also, does V4L2_CID_MPEG_VIDEOENC_AV1_HEADERS_WITH_FRAME refer all frames or just IDR frames?

I was looking for the equivalent to Video SDK’s NV_ENC_CONFIG_AV1.repeatSeqHdr

Hi,
We are checking the queries with our teams. Will update.

Some more queries/discrepancies:

What is the difference between V4L2_CID_MPEG_VIDEOENC_HW_PRESET_TYPE_PARAM and V4L2_CID_MPEG_VIDEOENC_CUDA_PRESET_ID? The former only seems to allow a smaller range of presets vs what NVENC is capable of. The latter suggests I can use a value 1-7, which is consistent with the 7 definitions in the desktop VIDEO SDK. So can I use the latter instead?

The sample code seems to use V4L2_CID_MPEG_VIDEOENC_HW_PRESET_TYPE_PARAM, but the comment underneath this definition in the file v4l2_nv_extensions.h suggests you have to fill out a v4l2_enc_hw_preset_type_param structure. However, the sample code does not do this, instead setting the control.value directly to the hw_preset_type, instead of the address of a filled-out structure. Is the sample code wrong, or the comment for how to use this control wrong?

Can I use V4L2_CID_MPEG_VIDEOENC_CUDA_TUNING_INFO? None of the samples or wrappers use it.

With the above two definitions, what is the meaning of “CUDA” in this context (e.g… CUDA_TUNING_INFO_). Does it only apply when I’m using CUDA surfaces or something? In the comments it refers to something called “CUVID Encoder”. What is a “CUVID Encoder” is it the same as NVENC, or something different?

What is the difference between V4L2_CID_MPEG_VIDEOENC_FORCE_IDR_FRAME and V4L2_CID_MPEG_MFC51_VIDEO_FORCE_FRAME_TYPE? Which should I use if I want to force an IDR frame, the first one, presumably? However, in the sample code there is a function enc->forceIDR() that uses V4L2_CID_MPEG_MFC51_VIDEO_FORCE_FRAME_TYPE, so now I’m not sure?

As a whole, the sample code seems pretty rough, if I may say so! Lots of duplicated blocks of code (for example huge chunks for reading the initial frames from a video file into the output planes, and then the same code pasted doing similiar/the same inside the encoder_proc_xxx functions)., Lots of evidence of lots of past changes, resulting in blocks like this (this is inside encode_unit_test)

bool stop_dqthread = false;
while (!stop_dqthread)
{
    // lots of code, but none of it using stop_dqthread!
}
stop_dqthread = true;

It’s hard to actually figure out what is old legacy code and what is the modern way to do things, but I’ll struggle on :)

The VIDEO SDK documentation was fantastic in comparison and always kept up to date with new releases etc.

Hi,
The design of hardware codec was different between dGPU and Jetson chips, so we have individual software stacks for each platform. Now hardware design is identical and we are unifying the interface. It is underway and this would take some time. So as of now, please follow the jetson_multimedia_api samples to develop your use-cases.

The document for jetson_multimedia_api is
Jetson Linux API Reference: Main Page | NVIDIA Docs

For setting preset level, please refer to the function in NvVideoEncoder.cpp:

int NvVideoEncoder::setHWPresetType(v4l2_enc_hw_preset_type type);

Thanks, that’s good to hear you plan to update the API.

I am aware of that function to set the preset, please re-read the first two paragraphs of my post.

Hi,

For nvEncGetSequenceParams, SPS PPS header is attached to the frame and not sent separately. You can parse it from the keyframe itself. Please refer to the patch for fetching SPS out:
Xavier AGX : Video encoding crash - #15 by DaneLLL

V4L2_CID_MPEG_VIDEOENC_AV1_HEADERS_WITH_FRAME control ID enables IVF headers on the AV1 frame. It begins with a 32-byte header and each frame contains a 12-byte header followed by the data.

We are checking other queries. Will update.

Thanks!

I’ve been trying to understand the samples, specifically what is legacy/unnecessary code, what I need in my application, etc.

I was looking at the encoder_unit_sample, so just using the low-level APIs and I noticed it opens the encoder in blocking mode, yet the dq_buffer function handles EAGAIN? The v4l2 documentation for VIDIOC_DQBUF is quite clear that this should only occur in non-blocking mode when there is no buffer available.

I got basic encoding working last night on my Jetson and found the same - even though I call open() on the encoder device without specifying any additional flags, thus defaulting to blocking mode, I still find that v4l2_ioctl (ctx->fd, VIDIOC_DQBUF, &v4l2_buf) frequently returns EAGAIN. So I just try again and a few iterations later it works. I haven’t investigated this further, as yet.

The sample code also seems to use an elaborate set of counters, locks and condition variables. Are these workarounds for something? From the few examples I’ve found online of v4l2 hardware video encoding, it should be sufficient just to call v4l2_ioctl with VIDIOC_QBUF and VIDIOC_DQBUF without any of that stuff surrounding it?

Is there some history behind all of this that I need to understand?

Unfortunately the sample code mostly isn’t commented so I have no way of knowing.

So I looked into this further, and it seems that DQBUF does block most of the time, but still seems to return EAGAIN a couple of times before it will actually give you the buffer.

Below is the trace if I queue an output buffer every 1s. As you can see it blocks for the 1s, but still returns EAGAIN. Only when you try and dequeue again do you get the buffer.

So I suppose the sample code where it allows up to 10 retries is a workaround for this issue. Where does the number 10 in the samples come from, is that just a “finger in the air, hope for the best” value or is it based on some logic?

But this isn’t expected behaviour is it?

21/12/23 09:28:54.690750020 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 999ms)
21/12/23 09:28:54.691199060 [default] DEBUG : VIDEO_ENCODER: Dequeued capture buffer after 0ms
Handling video encoder packet of size 14907.
21/12/23 09:28:54.691587682 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 0ms)
21/12/23 09:28:55.693010105 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 1000ms)
21/12/23 09:28:55.693470730 [default] DEBUG : VIDEO_ENCODER: Dequeued capture buffer after 0ms
Handling video encoder packet of size 17147.
21/12/23 09:28:55.693904473 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 0ms)
21/12/23 09:28:56.695025989 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 1000ms)
21/12/23 09:28:56.695553944 [default] DEBUG : VIDEO_ENCODER: Dequeued capture buffer after 0ms
Handling video encoder packet of size 17350.
21/12/23 09:28:56.696056107 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 0ms)
21/12/23 09:28:57.696653604 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 1000ms)
21/12/23 09:28:57.697314427 [default] DEBUG : VIDEO_ENCODER: Dequeued capture buffer after 0ms
Handling video encoder packet of size 15743.
21/12/23 09:28:57.697682729 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 0ms)
21/12/23 09:28:58.698392806 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 999ms)
21/12/23 09:28:58.698679696 [default] DEBUG : VIDEO_ENCODER: Dequeued capture buffer after 0ms
Handling video encoder packet of size 13280.
21/12/23 09:28:58.698811477 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 0ms)
q21/12/23 09:28:59.702971407 [default] DEBUG : VIDEO_ENCODER: Resource temporarily unavailable (waited 1003ms)
21/12/23 09:28:59.703294331 [default] DEBUG : VIDEO_ENCODER: Dequeued capture buffer after 0ms

How do I force the next output buffer queued to be encoded as an IDR frame?

First I tried this:

V4L2_CID_MPEG_VIDEOENC_FORCE_IDR_FRAME

But it has no effect. The comment also says “This control should be set after setting formats on both the planes and before requesting buffers on either plane.” which doesn’t make sense if the option is intended to force an IDR frame during encoding.

I saw in the sample code it uses:

V4L2_CID_MPEG_MFC51_VIDEO_FORCE_FRAME_TYPE

This forces an I-Frame, but the comment next to it states "Force I-frame on one of queued output plane buffer* so it’s not guaranteed to be the next one queued. I found this in practice too, it just picks a non-specific output buffer that has already been queued.

The official v4l2 documentation describes this option thus:

V4L2_CID_MPEG_MFC51_VIDEO_FORCE_FRAME_TYPE
(enum)
enum v4l2_mpeg_mfc51_video_force_frame_type -
Force a frame type for the next queued buffer. Applicable to encoders. Possible values are:

V4L2_MPEG_MFC51_FORCE_FRAME_TYPE_DISABLED Forcing a specific frame type disabled.
V4L2_MPEG_MFC51_FORCE_FRAME_TYPE_I_FRAME Force an I-frame.
V4L2_MPEG_MFC51_FORCE_FRAME_TYPE_NOT_CODED Force a non-coded frame.

Is it possible to tell the encoder that the next frame should be encoded as an I-Frame?

I can do this in the VIDEO SDK using the following, so I know the hardware is capable:

pic_params.encodePicFlags = NV_ENC_PIC_FLAG_FORCEIDR;

Hi,
So your suggestion is that in blocking mode, capture plane has to be blocking. It is wrong to be non-blocking and return EAGAIN. Thanks for pointing this out and we will check with our teams.

For forcing IDR frames, it does not set to specific frame in jetson_multimedia_api. So it may be 1-2 frame delay in getting IDR frame. We will check this to unify the behavior.

Hi there

Just following this up - are there equivalents in mmapi to these fields in Video SDK:

        config_hevc.hevcVUIParameters.videoSignalTypePresentFlag
        config_hevc.hevcVUIParameters.videoFormat
        config_hevc.hevcVUIParameters.videoFullRangeFlag
        config_hevc.hevcVUIParameters.colourDescriptionPresentFlag
        config_hevc.hevcVUIParameters.colourPrimaries
        config_hevc.hevcVUIParameters.transferCharacteristics
        config_hevc.hevcVUIParameters.colourMatrix
        config_hevc.hevcVUIParameters.chromaSampleLocationFlag
        config_hevc.hevcVUIParameters.chromaSampleLocationTop 
        config_hevc.hevcVUIParameters.chromaSampleLocationBot

…and some similar fields for AV1?

This allows this information to be written to the encoded bitstream and then things like an HLG HDR stream detected correctly by the player.

I can’t seem to find anything in either generic v4l2 or the mmapi customisations.

I can see there is something for decoding:

/**
 * Holds the video decoder output metadata for a frame.
 */
typedef struct v4l2_ctrl_videodec_outputbuf_metadata_
{
    /** Color primaries. */
    __u8 ucColorPrimaries;
    /** Transfer characteristics. */
    __u8 ucTransferCharacteristics;
    /** Matrix coefficients. */
    __u8 ucMatrixCoefficients;
    /** Boolean value indicating if \c FrameDecStats has valid contents. */
    __u32 bValidFrameStatus;
    /** Frame decode statistics. */
    v4l2_ctrl_videodec_statusmetadata    FrameDecStats;
    /** Codec specific metadata for the frame. */
    union {
        /** H.264 specific metadata. */
        v4l2_ctrl_h264dec_bufmetadata H264DecParams;
        /** H.265 specific metadata. */
        v4l2_ctrl_hevcdec_bufmetadata HEVCDecParams;
    }CodecParams;
} v4l2_ctrl_videodec_outputbuf_metadata;
/** @} */

But where can I set some of this info for a frame being passed to the encoder so that it gets written to the bitstream?

Hi,
On Jetson platforms, the parameters are filled in in low-level software stacks:

// When --insert-vui is set, these are set according to data format:
// BT.601, BT.601 ER, BT.709, BT.709 ER, REC2020
config_hevc.hevcVUIParameters.videoSignalTypePresentFlag
config_hevc.hevcVUIParameters.videoFormat
config_hevc.hevcVUIParameters.videoFullRangeFlag
config_hevc.hevcVUIParameters.colourDescriptionPresentFlag
config_hevc.hevcVUIParameters.colourPrimaries
config_hevc.hevcVUIParameters.transferCharacteristics
config_hevc.hevcVUIParameters.colourMatrix
// These are not filled in
config_hevc.hevcVUIParameters.chromaSampleLocationFlag
config_hevc.hevcVUIParameters.chromaSampleLocationTop 
config_hevc.hevcVUIParameters.chromaSampleLocationBot

Many thanks! Just need to figure out where I set it then, presumably you are implying that it should be set with the v4l2 output plane format where I currently set the pixel format, width, height etc.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.