tensorRT3 int8 calibrator: how to use read_calibration_cache() from Python?

I am using tensorRT3 int8 calibrator following this blog:
https://devblogs.nvidia.com/parallelforall/int8-inference-autonomous-vehicles-tensorrt

The blog is informative and helpful, except that read_calibration_cache() is not covered in detail. Could you please provide an example on the usage of read_calibration_cache()? How to return a pointer from the function?

Thanks!

Hi,

Please remember that INT8 feature is not available on TX2.

For read_calibration_cache, please check our document for details:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/topics/classnvinfer1_1_1_i_int8_calibrator.html

Thanks.

Thanks for your reply. But my trouble is with calling read_calibration_cache( ) from Python API. It generates error:

TypeError: SWIG director type mismatch in output value of type 'void const *'

I am not familiar with swig. How to write the read_calibration_cache() function? Could you provide an example on this?

Hi,

Have you checked our native INT8 sample?

It demonstrates the usage of readCalibrationCache and writeCalibrationCache, which can give you some hint:
/usr/src/tensorrt/samples/sampleINT8/sampleINT8.cpp

Thanks.

Got the same problem when read_calibration_cache() is called. The cpp sample mentioned above only shows how to implement this function in C++. The python error seems to be caused by the returned value with a wrong type.

Hi,

Python example can be found here:

It shows how to process image data and feed the calibrator . Hope it helps.
Thanks.

Hi,
In the example, read_calibration_cache() just returns None, which means always do not use the calibration cache. But if I want to return a python str type containing the calibration cache, then it reports the same error:

 TypeError: SWIG director type mismatch in output value of type 'void const *'

I also tried to return ctypes type, but still failed.
And for the input parameters of this function, need instructions on how to manipulate.
Could you provide more info about this python API? Thanks.

Hi,

Could you share an example to reproduce this issue?
Thanks.

The function is implemented like this:

def read_calibration_cache(self, length):
    f = open('CalibrationResults', 'rb')
    result = f.read()
    f.close()
    return result

I also don’t know how to handle input parameter “length”.

Hi,

Here is our document for your reference:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/topics/topics/pkg_ref/infer.html#tensorrt.infer.Int8Calibrator.read_calibration_cache
------
Parameters: length (*) – the length of the cached data, that should be set by the called function. If there is no data, this should be zero.
------

Thanks.

Hi,

I checked the document, it requires to return a pointer. But the problem is how to return a pointer in Python? I tried return a pointer with ctypes, but still got the same error. Can you provide an example of this Python function?

Hi,

For error - “TypeError: SWIG director type mismatch in output value of type ‘void const *’”
It’s recommended to refer to TensorRT sample - “sampleINT8” to implement your functions, such as readCalibrationCache.

For example:

class Int8LegacyCalibrator : public nvinfer1::IInt8LegacyCalibrator
{
public:
        ......
        const void* readCalibrationCache(size_t& length) override
        {
                mCalibrationCache.clear();
                std::ifstream input(locateFile("CalibrationTable"), std::ios::binary);
                input >> std::noskipws;
                if (mReadCache && input.good())
                        std::copy(std::istream_iterator<char>(input), std::istream_iterator<char>(), std::back_inserter(mCalibrationCache));
                length = mCalibrationCache.size();
                return length ? &mCalibrationCache[0] : nullptr;
        }
     .....

For python, since CalibrationTable is ready, please implement following function to read

def read_calibration_cache(self, length):
    ......

Thanks

Hi

My point is:

How to return a ‘void const *’ type in Python function “read_calibration_cache”.

This should depend on how the interaction implemented between Python and C++.
The only example I can found is in C++. Python should be different. Can you give an example of how to return data, in this Python function?

Hi,

We don’t have python sample currently.
You can implement read_calibration_cache and add the corresponding swig interface by your own.

Thanks.

Hi

Thanks for your advice.
Do I need to rewrite all about tensorrt.infer in TensorRT Python lib? Is “_nv_infer_bindings.so” in Python lib open source? Thanks.

You can implement your own function and also use the TensorRT function at the same time.

@AastaLLL is there a more rudimentary use case(in Python), I don’t need serialized plans etc, I just want to create the calibration_cache.bin file

Hi,

You can check this tutorial for information:
https://devblogs.nvidia.com/int8-inference-autonomous-vehicles-tensorrt/

Thanks.