Firstly, please run tlt-evaluate against your val dataset with the ngc pre-trained model(resnet18_peoplenet.tlt) you have downloaded, to check what the mAP result is.
Please refer to the spec in PeopleNet v1.0 unpruned model shows very bad results on COCO dataset - #11 by Morganh (especially pay attention to load_graph, it should be True, and for resnet34 model, the num_layers should be 34) and some comments in People Net - - #4 by ishan