For latest tao-converter, yes, you can use --layerPrecisions.
For example,
./tao-converter -e test_engine -k nvidia_tlt -p Input,1x3x544x960,1x3x544x960,1x3x544x960 --layerPrecisions=cls/Sigmoid:fp32,cls/Sigmoid_1:fp32,box/Sigmoid_1:fp32,box/Sigmoid:fp32,cls/Reshape_reshape:fp32,box/Reshape_reshape:fp32,Transpose2:fp32,sm_reshape:fp32,encoded_sm:fp32,conv_big_object:fp32 --layerOutputTypes=cls/Sigmoid:fp32,cls/Sigmoid_1:fp32,box/Sigmoid_1:fp32,box/Sigmoid:fp32,cls/Reshape_reshape:fp32,box/Reshape_reshape:fp32,conv_sm_object:fp32,sm_reshape:fp32,Transpose2:fp32,encoded_sm:fp32,convolution_output1:fp32,convolution_output2:fp32,convolution_output:fp32 --precisionConstraints=obey yolov3.etlt