NVEnc sample causes blue screen when attempting HEVC encode (Windows 7)

Hi there,

I’m trying to get the NVEnc SDK “NVEncoder” sample to run, and I find it causes display driver failures and a blue screen when using the HEVC codec with image widths greater than 4096. At 5120x1080 and 6144x1080, the display driver hangs and is re-started by Windows. At 7680x1080, the Timeout Detection & Recovery (TDR) fails and I get a blue screen.

The NVEncoder sample seems to run OK at HD resolution with H264 and HEVC codecs.

This is the first time I’ve tried using NVEnc, so I’m not sure where to look for a solution. There doesn’t appear to be all that much about NVEnc in these forums. Any pointers would be appreciated. System details etc. below.

Thanks
Al

Windows 7 x64
GeForce GTX 960
Driver version: 353.62
CUDA 6.5
NVEnc SDK 5.0.2
NVEncoder sample built using Visual Studio 2010 and Windows SDK 7.0A

Blue screen:

The computer has rebooted from a bugcheck.  
The bugcheck was: 
0x00000116 (0xfffffa8006f674e0, 0xfffff880049cf68c, 0x0000000000000000, 0x0000000000000002).

The VIDEO_TDR_ ERROR bug check has a value of 0x00000116. This indicates that an attempt 
to reset the display driver and recover from a timeout failed.

NVEncoder command line:

nvencoder -i Test.yuv -o Test.h265 -size 6144 1080 -codec 1

NVEncoder output:

Encoding input           : "Test.yuv"
         output          : "Test.h265"
         codec           : "HEVC"
         size            : 6144x1080
         bitrate         : 5000000 bits/sec
         vbvMaxBitrate   : 0 bits/sec
         vbvSize         : 0 bits
         fps             : 30 frames/sec
         rcMode          : CONSTQP
         goplength       : INFINITE GOP
         B frames        : 0
         QP              : 28
       Input Format      : YUV 420
         preset          : LOW_LATENCY_DEFAULT
  Picture Structure      : Frame Mode
         devicetype      : CUDA

OK I think I found some kind of answer: The GTX 960 supports encoding at up to 4096x4096. Sadly, the NVEncoder sample doesn’t check the device capabilities, and it appears that the driver doesn’t validate its inputs properly either.

Here’s a snippet to print the device capabilities - call this after m_pNvHWEncoder->Initialize:

void PrintCapabilities(CNvHWEncoder * pNvHWEncoder)
{
    if (!pNvHWEncoder) return;

    unsigned int encodeGUIDCount = 0;
    NVENCSTATUS nvStatus = pNvHWEncoder->NvEncGetEncodeGUIDCount(&encodeGUIDCount);
    if (nvStatus != NV_ENC_SUCCESS) {
        printf("Failed to get count of encode GUIDs. Status=%d\n", nvStatus);
        return;
    }

    GUID * encodeGUIDArray = new GUID[encodeGUIDCount];
    memset(encodeGUIDArray, 0, sizeof(GUID)* encodeGUIDCount);

    unsigned int encodeGUIDArraySize = 0;
    nvStatus = pNvHWEncoder->NvEncGetEncodeGUIDs(encodeGUIDArray, encodeGUIDCount, &encodeGUIDArraySize);
    if (nvStatus != NV_ENC_SUCCESS)
    {
        printf("Failed to get encode GUIDs. Status=%d\n", nvStatus);
        delete[] encodeGUIDArray;
        return;
    }

    assert(encodeGUIDArraySize <= encodeGUIDCount);

    for (size_t i=0; i<encodeGUIDCount; i++) {
        wchar_t strGuid[_MAX_PATH];
        StringFromGUID2(encodeGUIDArray[i], &strGuid[0], sizeof(strGuid));
        printf("Capabilities for GUID %u: %S\n", i, strGuid);

        struct CapabilityDescription
        {
            NV_ENC_CAPS capability;
            char const * name;
        };

        static CapabilityDescription allCapabilities[] = 
        {
            NV_ENC_CAPS_NUM_MAX_BFRAMES, "Maximum number of B-Frames supported",
            NV_ENC_CAPS_SUPPORTED_RATECONTROL_MODES, "Rate control modes supported",
            NV_ENC_CAPS_SUPPORT_FIELD_ENCODING, "HW support for field mode encoding",
            NV_ENC_CAPS_SUPPORT_MONOCHROME, "HW support for monochrome mode encoding",
            NV_ENC_CAPS_SUPPORT_FMO, "HW support for FMO",
            NV_ENC_CAPS_SUPPORT_QPELMV, "HW capability for Quarter pel motion estimation",
            NV_ENC_CAPS_SUPPORT_BDIRECT_MODE, "H.264 specific. HW support for BDirect modes",
            NV_ENC_CAPS_SUPPORT_CABAC, "H264 specific. HW support for CABAC entropy coding mode",
            NV_ENC_CAPS_SUPPORT_ADAPTIVE_TRANSFORM, "HW support for Adaptive Transform",
            NV_ENC_CAPS_SUPPORT_RESERVED, "Reserved enum field",
            NV_ENC_CAPS_NUM_MAX_TEMPORAL_LAYERS, "HW support for encoding Temporal layers",
            NV_ENC_CAPS_SUPPORT_HIERARCHICAL_PFRAMES, "HW support for Hierarchical P frames",
            NV_ENC_CAPS_SUPPORT_HIERARCHICAL_BFRAMES, "HW support for Hierarchical B frames",
            NV_ENC_CAPS_LEVEL_MAX, "Maximum Encoding level supported ",
            NV_ENC_CAPS_LEVEL_MIN, "Minimum Encoding level supported",
            NV_ENC_CAPS_SEPARATE_COLOUR_PLANE, "HW support for separate colour plane encoding",
            NV_ENC_CAPS_WIDTH_MAX, "Maximum output width supported",
            NV_ENC_CAPS_HEIGHT_MAX, "Maximum output height supported",
            NV_ENC_CAPS_SUPPORT_TEMPORAL_SVC, "Temporal Scalability Support",
            NV_ENC_CAPS_SUPPORT_DYN_RES_CHANGE, "Dynamic Encode Resolution Change Support",
            NV_ENC_CAPS_SUPPORT_DYN_BITRATE_CHANGE, "Dynamic Encode Bitrate Change Support",
            NV_ENC_CAPS_SUPPORT_DYN_FORCE_CONSTQP, "Forcing Constant QP On The Fly Support",
            NV_ENC_CAPS_SUPPORT_DYN_RCMODE_CHANGE, "Dynamic rate control mode Change Support.    ",
            NV_ENC_CAPS_SUPPORT_SUBFRAME_READBACK, "Subframe readback support for slice-based encoding",
            NV_ENC_CAPS_SUPPORT_CONSTRAINED_ENCODING, "Constrained Encoding mode support",
            NV_ENC_CAPS_SUPPORT_INTRA_REFRESH, "Intra Refresh Mode Support",
            NV_ENC_CAPS_SUPPORT_CUSTOM_VBV_BUF_SIZE, "Custom VBV Buffer Size support",
            NV_ENC_CAPS_SUPPORT_DYNAMIC_SLICE_MODE, "Dynamic Slice Mode Support",
            NV_ENC_CAPS_SUPPORT_REF_PIC_INVALIDATION, "Reference Picture Invalidation Support",
            NV_ENC_CAPS_PREPROC_SUPPORT, "Support for PreProcessing",
            NV_ENC_CAPS_ASYNC_ENCODE_SUPPORT, "Support Async mode",
            NV_ENC_CAPS_MB_NUM_MAX, "Maximum MBs per frame supported",
            NV_ENC_CAPS_MB_PER_SEC_MAX, "Maximum aggregate throughput in MBs per sec",
            NV_ENC_CAPS_SUPPORT_YUV444_ENCODE, "HW support for YUV444 mode encoding",
            NV_ENC_CAPS_SUPPORT_LOSSLESS_ENCODE, "HW support for lossless encoding",
        };

        int countOfAllCapabilities = sizeof(allCapabilities) / sizeof(allCapabilities[0]);

        for (int c=0; c<countOfAllCapabilities; c++) {
            CapabilityDescription desc = allCapabilities[c];
            NV_ENC_CAPS_PARAM caps;
            ZeroMemory(&caps, sizeof(caps));
            caps.version = NV_ENC_CAPS_PARAM_VER;
            caps.capsToQuery = desc.capability;
            int capabilityValue = 0;
            nvStatus = pNvHWEncoder->NvEncGetEncodeCaps(encodeGUIDArray[i], &caps, &capabilityValue);
            if (nvStatus == NV_ENC_SUCCESS) {
                printf("\t%s: %d\n", desc.name, capabilityValue);
            }
        }
    }

    delete [] encodeGUIDArray;
}

Here are the capabilities reported for a GTX 960:

Capabilities for GUID 0: {6BC82762-4E63-4CA4-AA85-1E50F321F6BF}
        Maximum number of B-Frames supported: 4
        Rate control modes supported: 63
        HW support for field mode encoding: 1
        HW support for monochrome mode encoding: 0
        HW support for FMO: 0
        HW capability for Quarter pel motion estimation: 1
        H.264 specific. HW support for BDirect modes: 1
        H264 specific. HW support for CABAC entropy coding mode: 1
        HW support for Adaptive Transform: 1
        Reserved enum field: 0
        HW support for encoding Temporal layers: 0
        HW support for Hierarchical P frames: 0
        HW support for Hierarchical B frames: 0
        Maximum Encoding level supported : 51
        Minimum Encoding level supported: 1
        HW support for separate colour plane encoding: 0
        Maximum output width supported: 4096
        Maximum output height supported: 4096
        Temporal Scalability Support: 0
        Dynamic Encode Resolution Change Support: 1
        Dynamic Encode Bitrate Change Support: 1
        Forcing Constant QP On The Fly Support: 1
        Dynamic rate control mode Change Support.    : 0
        Subframe readback support for slice-based encoding: 0
        Constrained Encoding mode support: 1
        Intra Refresh Mode Support: 1
        Custom VBV Buffer Size support: 1
        Dynamic Slice Mode Support: 1
        Reference Picture Invalidation Support: 1
        Support for PreProcessing: 0
        Support Async mode: 1
        Maximum MBs per frame supported: 65536
        Maximum aggregate throughput in MBs per sec: 983040
        HW support for YUV444 mode encoding: 1
        HW support for lossless encoding: 1
Capabilities for GUID 1: {790CDC88-4522-4D7B-9425-BDA9975F7603}
        Maximum number of B-Frames supported: 0
        Rate control modes supported: 63
        HW support for field mode encoding: 0
        HW support for monochrome mode encoding: 0
        HW support for FMO: 0
        HW capability for Quarter pel motion estimation: 1
        H.264 specific. HW support for BDirect modes: 0
        H264 specific. HW support for CABAC entropy coding mode: 1
        HW support for Adaptive Transform: 0
        Reserved enum field: 0
        HW support for encoding Temporal layers: 0
        HW support for Hierarchical P frames: 0
        HW support for Hierarchical B frames: 0
        Maximum Encoding level supported : 62
        Minimum Encoding level supported: 1
        HW support for separate colour plane encoding: 0
        Maximum output width supported: 4096
        Maximum output height supported: 4096
        Temporal Scalability Support: 0
        Dynamic Encode Resolution Change Support: 1
        Dynamic Encode Bitrate Change Support: 1
        Forcing Constant QP On The Fly Support: 1
        Dynamic rate control mode Change Support.    : 0
        Subframe readback support for slice-based encoding: 0
        Constrained Encoding mode support: 1
        Intra Refresh Mode Support: 1
        Custom VBV Buffer Size support: 1
        Dynamic Slice Mode Support: 1
        Reference Picture Invalidation Support: 1
        Support for PreProcessing: 0
        Support Async mode: 0
        Maximum MBs per frame supported: 65536
        Maximum aggregate throughput in MBs per sec: 983040
        HW support for YUV444 mode encoding: 0