What is the Map in pascal voc 07 dataset using this code? As i read your code, i find that you do not fix the BN parameters. The batchsize is small, following the paper FPN or Mask RCNN, you need to set the training and trainbale to False for the slim.batch_norm op. I am testing whether this is the problem for the bad generalization for mutiple objects.