The significance of builder.max_workspace_size.


What exactly is this parameter responsible for? If I set it to 1 << 30 does it mean it allocates 1GB GPU memory? Is it only for the engine building stage or for inference as well? I’m asking because it seems to me that whether I set it to 1GB or 4GB there is no difference in performance. Do you have any recommendations regarding this parameter?

I too would like to know the answer to this. I can say, that it is sometimes necessary to set for larger networks, or they will fail to build. The sampleGoogleNet.cpp sets this value (probably because it needs to). On one of my own networks, if I don’t set it, I fail with

Internal error: could not find any implementation for node blah, try increasing the workspace size with IBuilder::setMaxWorkspaceSize()
../builder/tacticOptimizer.cpp (1230) - OutOfMemory Error in computeCosts: 0



The maximum workspace limits the amount of memory that any layer in the model can use. It does not mean exactly 1GB memory will be allocated if 1 << 30 is set. During runtime, only the amount of memory required by the layer operation will be allocated, even the amount of workspace is much higher.