Memory errors inside encodeFromBuffer() function with quality = 100

nitzuas961 · August 13, 2024, 12:41pm

Hello,

The following code results in memory error, particularly invalid free() when quality = 100.

#include <vector>
#include <memory>
#include <stdint.h>
#include <random>
#include <ctime>

#include <NvJpegEncoder.h>

int main()
{
  int width = 512;
  int height = 512;

  std::srand(std::time(0));

  std::vector<uint8_t> bufY(width * height);
  for (int j = 0; j < width * height; j++) bufY[j] = std::rand() % 256;

  std::vector<uint8_t> bufU(width * height / 4);
  for (int k = 0; k < width * height / 4; k++) bufU[k] = std::rand() % 256;

  std::vector<uint8_t> bufV(width * height / 4);
  for (int l = 0; l < width * height / 4; l++) bufV[l] = std::rand() % 256;

  int quality = 100;

  auto m_encoder = std::unique_ptr<NvJPEGEncoder>(NvJPEGEncoder::createJPEGEncoder("jpenenc"));

  NvBuffer buffer(V4L2_PIX_FMT_YUV420M, width, height, 0);

  buffer.planes[0].data = bufY.data();
  buffer.planes[0].bytesused = width * height;

  buffer.planes[1].data = bufU.data();
  buffer.planes[1].bytesused = width * height / 4;

  buffer.planes[2].data = bufV.data();
  buffer.planes[2].bytesused = width * height / 4;

  uint64_t outBufSize = 3 * width * height / 2;

  uint8_t *outBufPtr;
  try
  {
    outBufPtr = new uint8_t [outBufSize];
  }
  catch(const std::bad_alloc& e)
  {
    printf("Allocation of memory for encoding failed \n");
    return 1;
  }

  if (m_encoder->encodeFromBuffer(buffer,
                                  JCS_YCbCr,
                                  &outBufPtr,
                                  outBufSize,
                                  quality) < 0)
  {
    printf("Encoding failed");
    delete[] outBufPtr;
    return 1;
  }

  delete[] outBufPtr;

  return 0;
}

However, empirically it was found out that when the buffer size is doubled (3 * width * height), errors disappear. In addition to that, when the code is compiled with the address sanitizer enabled, it seems that there is an unauthorized memory write inside encodeFromBuffer() (heap buffer overflow). Please see the logs below.

==112577==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xffff49a7d800 at pc 0xffff9778edc0 bp 0xffffebcc7730 sp 0xffffebcc77b8
WRITE of size 611703 at 0xffff49a7d800 thread T0
    #0 0xffff9778edbc in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790
    #1 0xffff76d37fdc  (/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so+0x3efdc)
    #2 0xffff4b9253f4  (/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so+0x43f4)
    #3 0xffff4b925690  (/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so+0x4690)
    #4 0xffff52d143b4 in NvMMLiteBlockDoWork (/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so+0x23b4)
    #5 0xffff52d27bdc  (/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so+0xbdc)
    #6 0xffff76d386a0 in jpegTegraEncoderCompress (/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so+0x3f6a0)
    #7 0xffff76cfec04 in jpeg_write_raw_data (/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so+0x5c04)
    #8 0xaaaaca21a394 in NvJPEGEncoder::encodeFromBuffer(NvBuffer&, J_COLOR_SPACE, unsigned char**, unsigned long&, int) /rootfs_arm64/usr/src/jetson_multimedia_api/samples/common/classes/NvJpegEncoder.cpp:267
    #9 0xaaaac96b69a0 in main main.cpp:343
    #10 0xffff52e16e0c in __libc_start_main ../csu/libc-start.c:308
    #11 0xaaaac98636c4  (/usr/bin/process+0x3f36c4)

0xffff49a7d800 is located 0 bytes to the right of 393216-byte region [0xffff49a1d800,0xffff49a7d800)
allocated by thread T0 here:
    #0 0xffff977f02b8 in operator new[](unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:107
    #1 0xaaaac96b6938 in main main.cpp:333
    #2 0xffff52e16e0c in __libc_start_main ../csu/libc-start.c:308
    #3 0xaaaac98636c4  (/usr/bin/process+0x3f36c4)

SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790 in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x200fe934fab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x200fe934fac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x200fe934fad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x200fe934fae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x200fe934faf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x200fe934fb00:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x200fe934fb10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x200fe934fb20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x200fe934fb30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x200fe934fb40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x200fe934fb50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==112577==ABORTING

After experimenting a lot I saw the same crashes in production and I suspect that there is a memory bug inside encodeFromBuffer, which I could neither explain, nor eliminate as I can not see the code.

Could you please help to fix the bug?

In addition: I have opened a topic sometime ago, which did not received an answer. Could you please reopen and help to progress that as well? The link to the topic: Segmentation fault in NvJPEGEncoder::encodeFromBuffer when encoder accessed from multiple threads - Jetson & Embedded Systems / Jetson AGX Orin - NVIDIA Developer Forums

The jetpack version is 5.1.3.

Thank you!

DaneLLL · August 14, 2024, 2:22am

Hi,
Please increase the output buffer size if you set quality=100. The default size may be insufficient for the condition.

nitzuas961 · August 14, 2024, 7:26am

Hi,

Thank you for the answer, I appreciate your response.
However, this is what I am already doing. I do not agree that it could be taken into use as a robust solution that is safe to rely on. If this is intended functionality, it should be documented and clearly stated that “the size of output buffer for the quality=100 should be X * width * height”. As it is not present in documentation or anywhere else, I see it as shaky workaround, which could get broken again at any point of time.

If it works now just by finetuning the buffer size, there is no guarantee that it is in principle correct and will be stable under any kind of boundary conditions. Also, if the output buffer size must be larger for quality=100, then what is the accurate formula ?

In addition about the second topic: unfortunately there is still no satisfiable answer under the thread Segmentation fault in NvJPEGEncoder::encodeFromBuffer when encoder accessed from multiple threads - Jetson & Embedded Systems / Jetson AGX Orin - NVIDIA Developer Forums.
There is advice to use synchronization via mutex, but it is not really relevant as the sample code provided in the first place uses mutex and in fact this is not enough – the code still segfaults, which again hints that there is serious issue inside NvJPEGEncoder. I would very much also appreciate if you could provide an answer about it.

Otherwise, these two issues seems like serious bugs.

DaneLLL · September 25, 2024, 5:10am

Hi,
It is possible the output size exceeds width*height*1.5 bytes if you generate the YUV420 like:

  std::vector<uint8_t> bufY(width * height);
  for (int j = 0; j < width * height; j++) bufY[j] = std::rand() % 256;

  std::vector<uint8_t> bufU(width * height / 4);
  for (int k = 0; k < width * height / 4; k++) bufU[k] = std::rand() % 256;

  std::vector<uint8_t> bufV(width * height / 4);
  for (int l = 0; l < width * height / 4; l++) bufV[l] = std::rand() % 256;

Please try with real-world scene.

system · October 23, 2024, 3:46am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
NvJPEGEncoder-encodeFromBuffer-tegra_multimedia_api Jetson AGX Orin mmapi	4	780	April 12, 2023
Jetson_multimedia_api samples ASAN failures Jetson AGX Orin mmapi , nvbugs	6	49	September 25, 2024
Segmentation fault in NvJPEGEncoder::encodeFromBuffer when encoder accessed from multiple threads Jetson AGX Orin encoder	3	190	May 14, 2024
NvJPEGEncoder::encodeFromBuffer trigger segmentation fault Jetson Orin Nano nvbugs , graphics	11	89	October 14, 2024
NvJPEGEncoder::encodeFromXxx - How to calculate buffer size? Jetson AGX Xavier	25	2634	July 25, 2019
NvVideoEncoder and the address-sanitizer Jetson TX2	9	802	October 18, 2021
How is the original image of NVBufSurface encoded as JPG via NVJPEG? Jetson Xavier NX graphics	2	1491	October 18, 2021
Argus - Memory Leak when retrieving NV12 Buffer Jetson AGX Orin camera	4	496	March 11, 2024
NVDecoder deallock on Jetson ORIN NX Jetson Orin NX decoder	5	30	February 3, 2025
How is the original image of NVBufSurface encoded as JPG via NVJPEG? DeepStream SDK	2	1297	October 12, 2021

Memory errors inside encodeFromBuffer() function with quality = 100

Related topics