A measurement system of gear dynamic transmission error based on optical encoder is built.The system first measures the rotation angles of the input and output shafts of a gear pair by two optical encoders. The collected pulse signals are then counted and interpolated to obtain angle sequences based on uniform time sampling. Finally,the dynamic transmission error of the gear pair can be obtained by comparing the angle sequences of two shafts. The nonlinear dynamics model of a gear system is built and it’s solved based on fourth-order Runge-Kutta method. The simulation results（ rotation angles） are sampled by simulating the optical encoder and data acquisition card and the influence of the resolution of the optical encoder and the time resolution of the data acquisition card on the measurement accuracy is studied. The simulation results show that in order to ensure the measurement accuracy for a gear system with a maximum speed of 1 500 r/min,the resolution of an optical encoder should be not less than 200 pulses per revolution and the time resolution of a data acquisition card should be not lower than 3. 3 × 10,-9, s. The gear transmission bench test also shows that the dynamic transmission error obtained by the measurement system has good repeatability,and its values are consistent with the noise phenomenon induced by the bench.