Grain size, characterized by a combination of grain length, width, and thickness, is one of the major determinants of yield in rice. The present study identified TATA-box binding protein-associated factor 2 (TAF2) as an essential component regulating transcription and determining grain size in rice. Map-based cloning showed that a G/T substitution in TAF2 resulted in a naturally occurring mutant called reduced grain size and plant height 1 (rgh1). The mutants, with weak edited rgh1 alleles, exhibited a small grain phenotype with reduced grain length and width, while the severe knockout mutant (rgh1-2s) was dwarf and completely sterile. Allelic test performed between rgh1 and several edited alleles confirmed that the mutation in TAF2 caused the rgh1 phenotype. GUS staining showed that TAF2 was mainly expressed in the vascular bundles of roots, stems, leaves, and grains. The cytological analysis revealed that reduced cell division in the glumes resulted in the small grain phenotype of rgh1. Further RNA-sequencing detected altered expression of genes involved in the basic biological processes in rgh1 mutant. These findings provide novel insights into the TAF2-mediated genetic mechanism regulating grain size in rice.