In order to study the torsional vibration of gear transmission system under slippage in high-speed train.A parametric vibration model of the gear transmission system is developed when the time-varying stiffness and the transmission error are considered.The nonlinear characteristic of selfexcited torsional vibration caused by wheel-rail creepage is analyzed.Numerical simulation results show that the time-varying mesh stiffness of meshing teeth pairs is the main cause of harmonic vibration of gear transmission system,when the wheel-rail creepage is over the best value,the self-excited vibration occurred and gear transmission system have the phenomenon of limited cycle in the same time,the self-excited vibration has no effect on the change of main frequency of the gear transmission system,but the harmonic response become more obvious,cause a strong vibration on the gear transmission system.