About inputs and outputs of a custom layer

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I’m trying to add a custom layer and have to implement virtual member function of IPluginExt, enqueue, whose parameters are like below.

virtual int enqueue(int batchSize, const void* const* inputs, void** outputs, void* workspace, cudaStream_t stream)

Foe example in the case where

  • the input and output ITensor have kFLOAT type
  • and this custom layer also have kFLOAT type,

does casting the inputs[0] to some pointer type which is not consistent with kFLOAT (like int*) violate
the strict-aliasing rules ?
In other words, does this fall under the following ?

If a program attempts to access the stored value of an object through a glvalue whose type is not similar ([conv.qual]) to one of the following types the behavior is undefined
http://eel.is/c++draft/basic.lval#11

From my understanding, this is legal as long as inputs and outputs are just allocated storage with no dynamic type.

Thank you!

2

As long as you follow the rules, using void* pointers is safe. But it is your responsibility to check that you follow the rules, otherwise the behavior is undefined.

See for example:

https://stackoverflow.com/questions/15745030/type-punning-with-void-without-breaking-the-strict-aliasing-rule-in-c99
https://en.cppreference.com/w/cpp/language/reinterpret_cast

3

Yes, that’s right, but that’s not what I meant.

To be more precisely, if TensorRT internally has the pointer to the storage, say, as float*, and casts it to void* and passes it to the enqueu function, it is illegal to convert inputs/outputs to any pointer type except for float* (, char*, unsigned char* or std::byte*).

I just want to make sure that this is not the case so that I can freely use inputs/outputs (,of course, as long as I stay within the rules as you said) !

I’ll show you a simple example below.

  1. This violates the rules.
// Inside the framework
float *data = allocate(); // If DataType is kFLOAT
enqueue((void *)data);

// My code
void enqueue(void *data) { int *input = (int *)data; }
  1. This follows the rules.
// Inside the framework
void *data = allocate(); // No matter what type DataType is
enqueue(data);

// My code
void enqueue(void *data) { int *input = (int *)data; }