A three- degree- of- freedom shaft- torsion coupling nonlinear dynamics model of a single-stage helical cylindrical gear transmission system is established,which included the nonlinear factors such as gear backlash,time- varying stiffness,transmission error and so on. A gear backlash function suitable for the helical cylindrical gear system is obtained by high order fitting the piecewise linear gear backlash function using in the spur cylindrical gear system. The results of the nonlinear dynamic response under two different gear backlash functions are obtained by employing the established dynamics equation of helical cylindrical gear system with variable step- size Runge- kutta method numerical solution. The influences of frequency on the bifurcation and chaotic characteristics under two different gear backlash functions are analyzed by using bifurcation diagrams,top Lyapunov exponent（ TLE） diagrams,phase portraits,Poincaré maps and FFT spectrums. The gear backlash function obtained by high degree fitting is more suitable for helical gear system by analyzing and comparing.