The 3D finite element model of gear transmission system of wind turbine is established. Combining experimental signal and analog signal on ADAMS,the effect of the fault which based on a single failure with gear tooth broken of fixed axis and coupling faults with crack tooth on fixed axis gear- planetary gear tooth broken on different points of transmission system is studied. The fixed axis gear with tooth broken have significant impact on each measure point and have the same frequency distribution characteristic. The changes of the speed does not affect the distribution of characteristic on frequency domain. The side frequency will appear at high- level mesh frequency and the medium speed level mesh frequency and the doubler. When the gear with coupling fault,the affect of fixed axis gear crack on the high frequency area of each measure point is obvious,the affect of planetary gear tooth broken on the low frequency area near the fault source is obvious,the amplitudes of mesh frequency and multiplier frequency of planet gear and the side frequency are obvious. The angular acceleration signal and experimental signal have the same distribution of characteristics.