Taking spur gear pair as the research object, a 6-DOF coupled bending-torsion nonlinear vibration model including time-varying meshing stiffness, comprehensive error, backlash and input torque is established. Based on the bifurcation diagram and Poincare map, the influence of backlash, input torque, their coupling effect and the support stiffness of the main and follower gear bearings on the vibration characteristics of the system is studied. The results show that under a certain input torque, the system transits from single-period response to chaos through bifurcation and catastrophe with the increase of backlash. Under a certain backlash, the system transits from chaos to one-period response through inverted bifurcation and catastrophe with the increase of input torque. When the input torque is large, the backlash has little effect on the system response. The stiffness of the follower gear bearing has a great influence on the vibration characteristics of the system.