The structural origins of bone toughness at the nanoscale are not completely understood. Therefore, we performed in situ scanning using atomic force microscopy during macroscopic mechanical testing of antler and bovine bone, to reveal the origins of the irreversible plastic deformation at the mineralized collagen fibril (MCF) array and MCF levels. We found that the plastic deformation behavior at the nanoscale level could be divided into two stages. The first stage of plastic deformation at the nanoscale level was characterized by slippage between the MCF arrays, which contained mineral aggregate grains with regular shapes under load. In the second stage of nanoscale plastic deformation, the MCFs broke through the bonds of the extrafibrillar mineral aggregate grains and exhibited interfibrillar slippage. These nanoscale plastic deformation behaviors may thus be the origins of stress whitening and irreversible plastic deformation. Thus, the findings in this study not only shed light on the plastic deformation mechanisms of MCF arrays and MCFs, but also provide structural and mechanistic insights into bioinspired materials design and mechanisms of relevant bone diseases.