The structure and working principle of a new type of non-circular planetary gear hydraulic motor are introduced. The pitch curve and tooth profile of each gear within non-circular planetary gear system are programmed. A torsional vibration model is built from the dynamic performance of non-circular planetary gear system. In the model,the time-varying meshing stiffness,meshing phase and backlash of the meshing gear pairs are considered. The dynamics differential equations are solved by using Runge-Kutta method. Taking the 6 ,8 elliptical planetary gear system as example,the time domain response of gear system is obtained and the influence of the backlash on the dynamic characteristic of the system is analyzed,a theoretical basis for vibration analysis of non-circular gear motor is provided.