A nonlinear dynamics model of the planetary gears-rotor system is established with a developed internal mesh stiffness embedded with ring gear stiffness. The torsional effect of rotor and pivotal parameters from gears including gear backlash， time-varying mesh stiffness and comprehensive transmission error are incorporated. Responses characteristic are analyzed with bifurcation diagrams， the largest Lyapunov exponents （LLE）， Poincaré maps and phase diagrams. The torsional vibration displacement response between gear and rotor is studied， and the influence law of torsional stiffness ratio between rotor shaft and drive shaft is analyzed. It is found that the system exhibits nonlinear characteristic， enters into chaotic motion through quasi-periodic bifurcation and period-doubling bifurcation， threshold values for stiffness ratios to avoid bifurcation and chaos are obtained. The research provides a reference of undesirable torsional behaviors in dynamic designing and vibration controlling for planetary gears-rotor systems in helicopter main reducer transmission.