how to support caffe tile layer?

hi, the prototxt i use is as follows:

layer {
  name: "comp0:Mul13169_caffe_1"
  type: "Tile"
  bottom: "comp0:Sigmoid_281"
  top: "comp0:Mul13169_caffe_1"
  tile_param {
    axis: 2
    tiles: 24
  }
}

and how can i add tensorrt support to this layer?

  1. particularly, how do i know the tile_param in the pluginFactory? below is the sample FCPluginFactory code and i dont see where i can parse the tile_param
    :
class FCPluginFactory : public nvcaffeparser1::IPluginFactoryExt, public nvinfer1::IPluginFactory
{
public:
	bool isPlugin(const char* name) override { return isPluginExt(name); }

	bool isPluginExt(const char* name) override { return !strcmp(name, "ip2"); }

    // Create a plugin using provided weights.
	virtual nvinfer1::IPlugin* createPlugin(const char* layerName, const nvinfer1::Weights* weights, int nbWeights) override
	{
		assert(isPluginExt(layerName) && nbWeights == 2);
		assert(mPlugin == nullptr);
        // This plugin will need to be manually destroyed after parsing the network, by calling destroyPlugin.
		mPlugin = new FCPlugin{weights, nbWeights};
		return mPlugin;
	}

    // Create a plugin from serialized data.
	virtual nvinfer1::IPlugin* createPlugin(const char* layerName, const void* serialData, size_t serialLength) override
	{
		assert(isPlugin(layerName));
        // This will be automatically destroyed when the engine is destroyed.
		return new FCPlugin{serialData, serialLength};
	}

    // User application destroys plugin when it is safe to do so.
    // Should be done after consumers of plugin (like ICudaEngine) are destroyed.
	void destroyPlugin() { delete mPlugin; }

    FCPlugin* mPlugin{ nullptr };
};
  1. how can i parse the layer according to type other than name?

That sample is specifically for adding support for exactly one type of plugin, their example “FCPlugin” class. I don’t think it’s the best example in the world because it doesn’t show how to parse more than one type of plugin. To support a new layer, and have tensorRT parse it, you’ll need to do to things:

  1. implement your plugin (a “layer” in caffe lingo)
  2. tell a factory how to identify it and parse it your plugin.

For 1, you’ll have to do some CUDA programming: either writing your own kernel, or using higher level APIs (various versions of cuda memcopy could help here).

For 2, what you need to do is implement the mandatory createPlugin functions for your plugin factory which identify tile layers (and possibly others), and then create them. If you’re using the caffe parser, I believe the layer’s “type” field is passed as the first argument layerName to createPlugin (though it may be the “name” field). This is how your plugin factory can decide what kind of layer to create. The next bit of trickiness regards the fields on a caffe Tile layer: the axis and number of tiles. I don’t believe there’s a way to have access to the caffe LayerParam inside of createPlugin – you might need to preprocess your proto to find those parameters and save them for when you call createPlugin. Fortunately though, tile layers have no weights or bias, so you don’t need to worry about those parameters.

Good luck,
Tom

Thanks for your help! However, i found something not going as i expected.
for a layer prototxt like blow:

layer {                                                                                            
  name: "Add19269_caffe_1"                                                                        
  type: "Tile"                                                                                    
  bottom: "Dimshuffle19267"                                                                
  top: "Add19269_caffe_1"                                                                         
  tile_param {                                                                                  
    axis: 0                                                                                       
    tiles: 2048                                                                                 
  }                                                                                            
}

i use the blowing code to parse it,

bool isPluginExt(const char* name) override { 
	printf("isPluginExt %s\n", name);
	return !strcmp(name, "Add19269_caffe_1");
}

virtual nvinfer1::IPlugin* createPlugin(const char* layerName, const nvinfer1::Weights* weights, int nbWeights) override
{
	printf("plugin name match %s\n", layerName);
	printf("nbWeights is %d\n", nbWeights);
}

and the output is:

isPluginExt Dimshuffle19267
isPluginExt Add19269_caffe_1
plugin name match Add19269_caffe_1
nbWeights is 0

so it seems like i still cant get type meta and the tile_param information.

i also use
Thanks.

Hi 13146012964,

I’m not sure what you mean by “type meta”, but it looks like you might be successfully identifying tile layers in your network (or at least certain tile layers). Congratulations! That’s an important step.

You say you don’t have access to the “tile_param” information, I’m assuming those are the parameters of your tile layer defined in your prototxt. Indeed, the nvcaffe_parser doesn’t have access to those directly in its plugin framework, so what can you do? What you can do is parse your network proto yourself ahead of time (using the Google Proto API) and identify the tile parameters present in it, and hold on to those (for example, in a map from layer name to tile parameters). When your NV-plugin factory identifies a tile layer, it can then look up the layer’s corresponding parameters, and subsequently build the layer.

Also, note that your createPlugin function as written has undefined behavior: it doesn’t return a value even though it says it returns an IPlugin*. It is mandatory to return a value of appropriate type here, or your program may crash, or corrupt its memory, or … who knows what else.

Cheers,
Tom