Because there are backlash friction and time-varying stiffness exist in gear transmission,the characteristics of gear pair has strong nonlinearity.When a gear has faults such as pitting,spalling and tooth breakage,there are often some nonlinear characteristics as super-harmonics or sub-harmonics along with vibration signal.These characteristics will bring some difficulty for fault diagnosis,so it is necessary to study the mechanism of nonlinear vibration under different gear fault degree.The Fourier expression is used to describe time-varying stiffness,and the problem of piecewise function stiffness mutation is avoided to some extent.The gear fault degree factor is used to depict gear fault degree,a nonlinear dynamics model for gear fault degree is established.And the experimental platform of the gear fault evolution is built.The simulation results and experimental results show that,with certain friction coefficient and rotating speed,the nonlinear vibration has direct relationship with rotating speed and gear friction coefficient and has no direct relationship with fault degree.