The coexisting periodic trajectory and its stability of a single- stage gear rotor transmission system with 3 degree- of- freedom are studied by using the method of CPNF（ Continue- Poincare- Newton-Floquet）. The finite difference method is used to approximate instead of the Jacobi matrix of non- smooth system,and the defect that the nonlinear dynamic system must be smooth of the CPNF method is improved.Through calculation results of improved CPNF method comparing with the results of numerical calculation for example,the improved CPNF method is verified to be effective. The coexisting periodic motion of the gear rotor system is studied by using the method of improved CPNF,the stability of each periodic motion is determined at the same time. Through the CPNF,the stability of the periodic trajectory under different speeds is determined,the bifurcation characteristic of the gear rotor system under certain dimensionless rotational speed is studied.The results show that a nonlinear gear rotor system with certain parameters may have several coexisting stable or unstable periodic trajectories. Moreover,the motion of the system could also evolve into chaos by way of period- doubling bifurcation as the speed decreases in 1. 54 ,1. 42.