The profile of the gear in internal helical gear is modified to realize point to point contact. The transmission coordinate system is built to deduce the tooth surface equation,the nonlinear meshing equation is built based on meshing theory. The relative velocity and normal vector of the reference point are calculated,and the numerical solution of nonlinear meshing equations is obtained with newton iteration method. The numerical solution is substituted into the mesh equation to solve the transmission error,tooth contact trace and contact ellipse,consequently realized the mesh simulation of the internal helical gears which tooth surface is carried out.