Rotational speed stability is an important evaluation indicator of the performance of a hydro-viscous clutch (HVC). To improve the rotational speed stability of HVCs in mixed lubrication and the running condition of the friction pairs, the speed stability of an HVC in mixed lubrication was studied. To this end, the friction coefficients of both copper-based and paper-based friction pairs were experimentally tested using an MM1000-III wet friction machine. Theoretically, a torsional vibration model of the system is presented. The phase plane analysis method is applied to evaluate the stability of the torsional vibration model, where a critical negative gradient (CNG) is defined. The results show that the friction coefficient in mixed lubrication is an important parameter for the stability of the rotational speed. The system will be unstable when the negative gradient of the friction coefficient-slip speed is larger than the CNG. According to the definition of the CNG, suggestions regarding choice of friction pairs are made to improve the rotational speed stability of an HVC in mixed lubrication.