In order to improve the meshing performance of the internal helical gear transmission pair, a topographic modification method of the internal helical gear is proposed. According to the principle of tooth profile and helix modification, the equation of internal helical gear modification tooth surface is deduced, the tooth profile and longitudinal modification coefficients are preset, the contact analysis (TCA) model of internal helical gear pair is constructed by using the gear meshing principle, and the transmission error under different modification coefficients is obtained, the transmission error under different modification coefficients and different loads is obtained by the finite element simulation analysis. The results show that different modification coefficients should be selected under different operating conditions in order to meeting the operational requirements, and this method provides a reference for the design of the topologically modified internal helical gear for forming grinding.