Grain size is one of the most important factors affecting rice grain quality and yield, and attracts great attention from molecular biologists and breeders. In this study, we engineered a CRISPR/Cas9 system targeting the miR396 recognition site of the rice GS2 gene, which encodes growth-regulating factor 4 (OsGRF4) and regulates multiple agronomic traits including grain size, grain quality, nitrogen use efficiency, abiotic stress response, and seed shattering. In contrast to most previous genome editing efforts in which indel mutations were chosen to obtain null mutants, a mutant named GS2^E carrying an in-frame 6-bp deletion and 1-bp substitution within the miR396-targeted sequence was identified. GS2^E plants showed increased expression of GS2 in consistent with impaired repression by miR396. As expected, the gain-of-function GS2^E mutant exhibited multiple beneficial traits including increased grain size and yield and bigger grain length/width ratio. Thousand grain weight and grain yield per plant of GS2^E plants were increased by 23.5% and 10.4%, respectively. These improved traits were passed to hybrids in a semi-dominant way, suggesting that the new GS2^E allele has great potential in rice improvement. Taken together, we report new GS2 germplasm and describe a novel gene-editing strategy that can be widely employed to improve grain size and yield in rice. This trait-improvement strategy could be applied to other genes containing miRNA target sites, in particular the conserved miR396-GRF/GIF module that governs plant growth, development and environmental response.