I got my code working.
It looks like the implementation of jpeg_mem_src() is a bit broken in libnvjpeg.so. It creates a jpeg_source_mgr that is invalid for use with jpeg_read_raw_data(), causing segfaults as jpeg_read_raw_data() tries to copy jpeg data to be decoded.
So I had to manually create my own jpeg_source_mgr and implement my own init_source(), fill_input_buffer(), skip_input_data(), resync_to_restart(), and term_source() functions.
Unfortunately, the performance is still not great. It still takes 30-35 ms to decode a 1920*1080 image, whereas I know that the gstreamer plugin can do much faster than that by leveraging NVMM.