Germplasm conserved in gene banks is underutilized, owing mainly to the cost of characterization. Genomic prediction can be applied to predict the genetic merit of germplasm. Germplasm utilization could be greatly accelerated if prediction accuracy were sufficiently high with a training population of practical size. Large-scale resequencing projects in rice have generated high quality genome-wide variation information for many diverse accessions, making it possible to investigate the potential of genomic prediction in rice germplasm management and exploitation. We phenotyped six traits in nearly 2000 indica (XI) and japonica (GJ) accessions from the Rice 3K project and investigated different scenarios for forming training populations. A composite core training set was considered in two levels which targets used for prediction of subpopulations within subspecies or prediction across subspecies. Composite training sets incorporating 400 or 200 accessions from either subpopulation of XI or GJ showed satisfactory prediction accuracy. A composite training set of 600 XI and GJ accessions showed sufficiently high prediction accuracy for both XI and GJ subspecies. Comparable or even higher prediction accuracy was observed for the composite training set than for the corresponding homogeneous training sets comprising accessions only of specific subpopulations of XI or GJ (within-subspecies level) or pure XI or GJ accessions (across-subspecies level) that were included in the composite training set. Validation using an independent population of 281 rice cultivars supported the predictive ability of the composite training set. Reliability, which reflects the robustness of a training set, was markedly higher for the composite training set than for the corresponding homogeneous training sets. A core training set formed from diverse accessions could accurately predict the genetic merit of rice germplasm.