In order to study the influence of tooth surface microstructure on gear motion stability,the fractal theory is introduced to realize the simulation of the shape of the cylindrical straight gear tooth surface through the characteristic data of the real tooth surface. With the help of W-M function,M-B fractal contact model and Hertz contact formula,the stiffness and damped mathematical model considering the uneven surface of gear surface is established. The influence of different machining methods on the normal contact stiffness and damping of the tooth surface is analyzed,MATLAB and ADAMS are used to study the influence of normal contact stiffness and damping of tooth surface on the vibration characteristics of gear meshing. The results show that the microstructure of tooth surface affects the stability of gear motion. The machining method affects the meshing stiffness and damping of gears. Compared with grinding and milling,the normal contact damping of electrochemical finishing method is more larger,the method is more effective to reduce the vibration of gears and improve the stability of gear.