Gstreamer nvjpegenc plugin fails at runtime

Im running a TX2 with JP4.4, gstreamer 1.14.5, cuda 10.2.
Im trying to make a gstreamer pipeline to decode an h264 stream and send to an appsink the frames so I can use them in other apps.
The nvjpegenc seems to be using different libjpeg headers and when I try to run the program it throws the following error:

JPEG parameter struct mismatch: library thinks size is 584, caller expects 728

The pipeline is the following

"appsrc name=visualsrc do-timestamp=true is-live=true "
"! queue "
"! video/x-h264, width=1920, height=1080, framerate=30/1, stream-format=(string)byte-stream, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive "
"! h264parse "
"! nvv4l2decoder enable-error-check=true enable-max-performance=true "
"! nvjpegenc quality=100 "
"! nvvidconv "
"! video/x-raw, width=1920, height=1080 "
“! appsink name=framesink”;

I managed to run no problem with a multifile sink(instead of the appsink) writing to disk all of the encoded jpegs.

"! multifilesink name=multisink max-files=100 location=/mnt/ssd/frames/frame%d.jpg ";

Also the pipeline works using the SW encoder jpegenc and the appsink but I would really like to use the hardware to do this task, as many solutions around just point to use the SW encoder. SW encoding basically uses an entire CPU core while using the HW encoder brings it down to 5/10% CPU usage(1 core).

Thanks for the help!

Since it loads in OpenCV, nvjpegenc cannot be used due to conflicts. Please refer to explanation in
OpenCV with libnvjpeg