Because of the output response of the wind turbine transmission system caused by the gear fault is not stable,the dynamic performance of the transmission system in fault state is studied. The transmission system dynamics models under normal condition and gear fault condition are established respectively by considering the influence of time-vary stiffness and gear crack of helical gear,and the output responses of two systems are analyzed and compared. The results show that the output engaging force,torsional vibration displacement and torsional vibration velocity are relatively large and the irregular change trends are presented under the fault gear condition. The impact vibration is large at some simulation time points. The theoretical reference for the fault diagnosis and early fault prediction of the wind turbine transmission system is provided by the research results.