How to train model in Tensorflow for multi class Object Detection using large MS COCO Dataset?

I am trying to train my model on custom dataset using this https://github.com/tensorflow/models/tree/master/research/object_detection as a reference. Basically I am having Acer Nitro 50 Desktop with system configuration Processor: Intel® Core™ i5-8400 CPU @ 2.80GHz × 6, Graphics: GeForce GTX 1050/PCIe/SSE2 (2 GB), Memory(RAM): 8GB DDR4 Memory

I am working with tensorflow 1.12.0 gpu | bazel 0.15.0 | python 3.5 | GCC 4.8 | cudnn 7 | Cuda 9.0 to train a faster_rcnn_inception_v2_coco model on my custom ms coco dataset with 10.4 GB(65000 images) of training data and 533.4 MB(3300 images) of validation data for object detection for 200k epochs(num_steps it will be training at 600 x 1024 default resolution which is set in faster_rcnn_inception_v2_coco). I am training and validation model on 8 classes. So when I am training the model the accuracy(map) is not increasing and loss is not decreasing after a while. After the completion of the training successfully I ran the model on various images and noticed couple of things. I have couple of questions

  1. Less accuracy
  2. In some images objects does not get detected at all no bounding boxes. while in some multiclass object detection is spot on perfectly all objects gets detected but with less accuracy around 60 - 75.
  3. From second point above If I train a separate model with less images and less number of classes(3 or 4) it works well but with decent amount of accuracy around 75 – 95

As you can see the above images my model is not training effectively with total loss and learning rate being inconsistent. Hence map is not going up resulting into less accuracy. It would be really helpful if anyone can guide me. I am trying to figure out this issue from a while. I am not getting any errors or warnings.

Here are the changes required to train the model in file faster_rcnn_inception_v2.config

# Faster R-CNN with Inception v2, configuration for MSCOCO Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.
model {
  faster_rcnn {
    num_classes: 8
#Default configuration for image_resizer no changes made in this function
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 600
        max_dimension: 1024
      }
    }
    feature_extractor {
      type: 'faster_rcnn_inception_v2'
      first_stage_features_stride: 16
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        scales: [0.25, 0.5, 1.0, 2.0]
        aspect_ratios: [0.5, 1.0, 2.0]
        height_stride: 16
        width_stride: 16
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 300
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        use_dropout: false
        dropout_keep_probability: 1.0
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.0
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 300
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}

train_config: {
  batch_size: 1
  optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.0002
          schedule {
            step: 900000
            learning_rate: .00002
          }
          schedule {
            step: 1200000
            learning_rate: .000002
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "models/model.ckpt"
  from_detection_checkpoint: true
  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the COCO dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.
  num_steps: 200000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: "data/dataset_tools_train.tfrecord"
  }
  label_map_path: "data/label.pbtxt"
}

eval_config: {
  num_examples: 3300
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "data/dataset_tools_val.tfrecord"
  }
  label_map_path: "data/dataset_tools_val.tfrecord"
  shuffle: false
  num_readers: 1
}