Early dental caries detection by endoscope can prevent complications, such as pulpitis and apical infection. However, automatically identifying dental caries remains challenging due to the uncertainty in size, contrast, low saliency, and high interclass similarity of dental caries. To address these problems, we propose the Global Feature Detector (GFDet) that integrates the proposed Feature Selection Pyramid Network (FSPN) and Adaptive Assignment-Balanced Mechanism (AABM). Specifically, FSPN performs upsampling with the semantic information of adjacent feature layers to mitigate the semantic information loss due to sharp channel reduction and enhance discriminative features by aggregating fine-grained details and high-level semantics. In addition, a new label assignment mechanism is proposed that enables the model to select more high-quality samples as positive samples, which can address the problem of easily ignored small objects. Meanwhile, we have built an endoscopic dataset for caries detection, consisting of 1318 images labeled by five dentists. For experiments on the collected dataset, the F1-score of our model is 75.6%, which out-performances the state-of-the-art models by 7.1%.