For nano there is 4GB RAM, only less than 2GB could be assigned to the GPU side.
On jetson, tensorRT engine with FP16 mode is preferred to do the inference.
If you stick in with Tensorflow. There is also a parameter can be used to fix the GPU allocation.
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config, ...)