Potential int32 overflow in nvjpeg causing nvjpegEncodeImage failure for large images

Accelerated Computing GPU-Accelerated Libraries
, ,
1

Hi NVIDIA team,

I am running into an unexpected behavior when encoding large images using nvJPEG, and I would appreciate your help in confirming if this might be a bug.

When the image dimensions increase slightly beyond a certain point, nvjpegEncodeGetBufferSize returns an unusually large size_t value. This then seems to cause nvjpegEncodeImage to fail with error code 6 (NVJPEG_STATUS_EXECUTION_FAILED) and destroyed other resource context (like TensorRT).

Based on the returned value, I suspect there might be a 32-bit signed integer overflow happening internally during the buffer size calculation. For example, when encoding an image of 26400 x 13272, the required size calculates as expected (~2.1 GB). However, when the width is slightly increased to 27200 x 13272, the required size jumps to 18446744071580576768.

Interestingly, this massive value corresponds to 0xFFFFFFFF8122C000 in hex, which behaves exactly like a negative 32-bit integer (-2128437248) that has been cast to a 64-bit unsigned size_t.

Request for Confirmation

Could you kindly help confirm if this is indeed a bug caused by integer overflow, or if there is a documented maximum resolution limit for nvjpegEncodeImage that I might have overlooked?

If it is an overflow issue, would it be possible to consider updating the internal calculations to use 64-bit integers in a future release to better support high-resolution images?

Thank you for your time and support!

Minimum reproducible example

#include "nvjpeg.h"
#include "logger.h"
#include <iostream>
#include <cstdio>
#include <cstring>
#include <vector>

bool encode_device_to_jpeg(
	NVJPEGBuffer &nv,
	const uint8_t *d_image,
	int width, int height, int channels,
	std::vector<char> &out_file_data,
	cudaStream_t stream)
{
	nvjpegEncoderState_t encoder_state;
	nvjpegEncoderParams_t encoder_params;

	nvjpegEncoderStateCreate(nv.nv_handle, &encoder_state, stream);
	nvjpegEncoderParamsCreate(nv.nv_handle, &encoder_params, stream);
	nvjpegEncoderParamsSetQuality(encoder_params, 100, stream);
	nvjpegEncoderParamsSetOptimizedHuffman(encoder_params, 1, stream);
	nvjpegEncoderParamsSetSamplingFactors(encoder_params, NVJPEG_CSS_444, stream);

	nvjpegImage_t img_desc = {0}; 
	img_desc.channel[0] = (unsigned char *)d_image;
	img_desc.pitch[0] = width * channels;

	size_t max_stream_length = 0;
	nvjpegStatus_t get_size_status = nvjpegEncodeGetBufferSize(
		nv.nv_handle, encoder_params, width, height, &max_stream_length);

	if (get_size_status == NVJPEG_STATUS_SUCCESS)
	{
		char buf[128];
		snprintf(buf, sizeof(buf), "nvjpegEncodeGetBufferSize required size: %zu bytes for image (%d x %d)", max_stream_length, width, height);
		CU_LOG_INFO(buf);
	}

	nvjpegStatus_t status = nvjpegEncodeImage(
		nv.nv_handle, encoder_state, encoder_params,
		&img_desc,
		NVJPEG_INPUT_BGRI,
		width, height,
		stream);

	if (status != NVJPEG_STATUS_SUCCESS)
	{
		char buf[128];
		snprintf(buf, sizeof(buf), "nvjpegEncodeImage failed with status %d", static_cast<int>(status));
		CU_LOG_ERROR(buf);
		nvjpegEncoderParamsDestroy(encoder_params);
		nvjpegEncoderStateDestroy(encoder_state);
		return false;
	}

	size_t length = 0;
	nvjpegEncodeRetrieveBitstream(nv.nv_handle, encoder_state, NULL, &length, stream);

	out_file_data.resize(length);
	nvjpegEncodeRetrieveBitstream(nv.nv_handle, encoder_state, (unsigned char *)out_file_data.data(), &length, stream);

	cudaStreamSynchronize(stream);

	nvjpegEncoderParamsDestroy(encoder_params);
	nvjpegEncoderStateDestroy(encoder_state);

	return true;
}

Relevant log output

Below I post logs of two images demonstrating the sudden jump in the required buffer size and the subsequent failure.

[2026-06-25 15:40:25.613] [info] [testNVJPEG.cpp:34] nvjpegEncodeGetBufferSize required size: 2102286848 bytes for image (26400 x 13272)
[2026-06-25 15:41:20.208] [info] [testNVJPEG.cpp:34] nvjpegEncodeGetBufferSize required size: 18446744071580576768 bytes for image (27200 x 13272)
[2026-06-25 15:41:22.589] [error] [testNVJPEG.cpp:48] nvjpegEncodeImage failed with status 6

Environment

  • CUDA Version: [cuda toolkit 12.9]
  • GPU Model: [RTX 4060, RTX 4070S, A800 40GB]
  • OS: [win11]
  • Driver Version: [nvJpeg 12.4.0.16]