DRIVE OS Version: 7.0.3
Issue Description: We receive images encoded with H265 encoding and want to decode them. How to decode data on Thor?
The Documentation lists a decoder module, but that has only a handle defined, while the encoder module has functionality implemented.
How should we decode H265 Data on Thor? Should we use some other library?
Hello @SivaRamaKrishnaNV,
i was already investigating this example while i waied for your reply. Indeed, this example does roughly what we want with just nvmedia, but itās videodemo.c file is almost 2000 lines long and very complicated to understand. It has few comments and the code and variable names seem not very intuitive.
For example, the examples configures a struct with many callback functions, and itās not really clear to me how the Decode_orig function (seemingly where the magic happens) decodes the encoded data or receives it back from the library.
Also, do we need to āparseā the encoded data with the NvMediaParser API? Which parts of the example do we actually need, considering that we want to decode just H265 data and display it? Will implementing it with nvmedia be possible if we also need to use driveworks for displaying the images?
So yes, it helps, but itās still not really clear how to implement decoding for our application (in short: receive H265 data, decode and display it).
Is it encoded using NvMedia APIs? If you have h264 video, we have DW sample to decode and display using Camera Seek Sample ā NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide . H265 is deprecated on DW.
We do the encoding on another system with the sensor serialization API of driveworks SDK 6.0.6, then send the data over the network.
H265 is deprecated on DW.
Where is this documented? We didnāt see anything of the sort. Which encoding format do you recommend instead?
The Camera Seek Sample may do roughly what we need, but we only have the compiled binary of it, no source code to work with.
I doubt the compatibility of data encoding across DW versions.
I remember seeing deprecation messages with recorder tool like in Running simulation using recorded data from gmsl cameras . We recommend to use mp4 format with DW
Regarding your last answer:
I doubt the compatibility of data encoding across DW versions.
Surely, the encoding follows the standard and does not change based on implementation? It would be weird if different versions or implementations produced different encoded data, since I think codecs like H265 and AV1 are standardized.
We recommend to use mp4 format with DW
Since mp4/MPEG-4 is just a container format that also contains encoded video data (like H265), i am confused by this statement. If H265 is depreciated and you recommend mp4, then to which format will the video actually be encoded while being embedded into the MPEG-4 container?
That being said, I am still trying to get decoding to work, by using nvmedia, but have not much more than draft api documentation and a very large and seemingly complicated example source code that does something similar. Decoding is crucial for our project, but itās not looking too good right now.
MP4 format uses h264 encoding .
Hello.
Iām still trying to get decoding to work. I have adapted much of the nvmedia decoding sample for our own module. It compiles now, but I canāt get it to work. The error for now is that the parser creation with NvMediaParserCreate simply returns NULL without explanation.
According to the Documentation, I need to meet no precondition. I have set the parameters as follows:
// the struct is filled with callbacks. All these functions exist and have the correct signature.
NvMediaParserClientCb NV_PARSER_CALLBACKS = {&nv_cb_BeginSequence,
&nv_cb_DecodePicture,
&nv_cb_DisplayPicture,
&nv_cb_UnhandledNALU,
&nv_cb_AllocPictureBuffer,
&nv_cb_Release,
&nv_cb_AddRef,
NULL,
NULL,
NULL,
NULL,
&nv_cb_GetBackwardUpdates,
NULL};
DecodeManager::DecodeManager() {
// abbreviated code
this->nv_parser_params.pClient = &NV_PARSER_CALLBACKS;
this->nv_parser_params.pClientCtx = this;
this->nv_parser_params.eCodec = NvMediaVideoCodec::NVMEDIA_VIDEO_CODEC_HEVC;
// NOTE: keep default values for some weird values
this->nv_parser_params.uReferenceClockRate = 0;
this->nv_parser_params.uErrorThreshold = 0;
/** Optional external sequence header data */
this->nv_parser_params.pExternalSeqInfo = NULL;
this->nv_parser = NvMediaParserCreate(&this->nv_parser_params);
if (nullptr != this->nv_parser) {
throw std::runtime_error(
"NvMediaParserCreate was unsuccessfull and returned a null pointer");
}
(the above is part of a larger module. Please note that the issue does not seem to be that the code is using C++ classesā members, as the same behavior occurs when I have the parser parameters as a local variable.)
Still, it always returns NULL, and I do not understand why. I feel like Iām writing my own decoder by now, even though we thought thats something provided by nvidia.
Hello @SivaRamaKrishnaNV. We are still trying to get decoding to work, but get NULL after using NvMediaParserCreate. Advise would be appreciated.
Hello,
we are still hoping for guidance on this topic. Currently, we have the problems that the nvmedia NvMediaParserParse function accepts our encoded data, but never actually calls the registered callbacks to do the decoding.
We are working with H265 encoded images, sent to us and encoded by another driveos device.
Dear @christoph.scherr ,
Do you want to know the cases when NvMediaParserCreate can return NULL?
Also, is the other device also using same DOS version where encoding is performed?
Do you want to know the cases when NvMediaParserCreate can return NULL?
This is not our issue anymore, we got that to work. Right now, the problem is that the NvMediaParserParse does not call the callbacks that do the actual decoding, defined by setting the NvMediaParserClientCb in the parameters. We have the BeginSequence callback being called sometimes, but arenāt sure why it sometimes doesnāt get called.
Another issue is going to be how to display the images once we have them decoded. Is there a way to get a driveworks dwImageHandle_t somehow after decoding with nvmedia? If not, what other option do you recommend for displaying the decoded image data?
Also, is the other device also using same DOS version where encoding is performed?
No, they are different. But since the other device produces valid encoding, that should not be a problem. When we save the received frames to a file, we can replay the h265 video with a program like VLC media player.
Dear @christoph.scherr ,
As I understand from the thread, you have encoded the data using DW and sending it over network to other machine for decoding. If so, you can use DW SAL APIs to read the encoded stream and display directly. DW sample_replay demonstrates it already. Did you check it?
Encoding using DW and decoding using NvMedia does not work.
You understand our usecase correctly. We are only using nvmedia because, as far as we know, driveworks is unable to do decoding and displaying. If there is an easier method than using nvmedia, we would be very happy to learn about it.
I cannot find the āsample_replayā sample. Where is it? Can you link them?
And about the DW SAL API: Can you explain how to use it to decode and display H265 data?
In addition, I donāt understand why DW encoded H265 data would not be compatible with nvmedia, Since both support the standardized H265 format.
Dear @christoph.scherr ,
We have DriveWorks SDK Reference: Camera Replay Sample available in DOS 6.x release for Orin. But If you are using Thor, we have Camera Seek Sample ā NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide to check replaying. But it does not seem to support H265.
Please give a try with nvm_ide_sci sample to confirm if it working before plan to use NVMedia Decoder APIs.
The camera replay sample at least explicitly supports H265 according to the link to documentation you provided:
The Camera Replay sample demonstrates H.264/265/MP4 playbac
However, neither of these samples seem to have source code available, so they unfortunately arenāt of much help to us.
Dear @christoph.scherr ,
Please check /usr/local/driveworks/samples folder on docker for DW samples. Note that few samples are removed in DOS 7.0.3 which are available in DOS 6.0.10.
I found the samples, and they are indeed helpful. But one question: Is it possible to decode images from memory, rather than those that are stored in a file? Having to store the images into some file after we receive them only to load them immediately again would waste a lot of performance.
Did you test camera replay sample and is working for your use case?