Based on the system dynamics,geometric property of involute helical gear,the transmission error analysis of helical gear is carried out by using the software ABAQUS. A method to determine the tooth profile modification curves and modification quantity according to the gear transmission error is proposed. The transmission error of gear pair can be effectively reduced by using the method and an important research method for reducing the gear system vibration and noise control is provided.