Based on the consideration of various parts of spur gear,the three degree of freedom dynamics model of gear transmission system is established. With the help of the mathematical software MATLAB,the gear dynamics equation constructed by using the Runge- kutta method is solved,the dynamics response curve is got,and the bifurcation and stability of the gear transmission system under different excitation frequency are analyzed. Through the analysis found the complex bifurcation structure and general rule are exist in the strongly nonlinear gear system,a reference for further study of gear system nonlinear dynamics is provided.