Taking the gear pair meshing stiffness,meshing damping and transmission error into consideration,the six degree of freedom pure torsion model differential equation of multi- gear transmission is set up based on lumped parameter method,the vibration accelerated speed of each component is obtained by adopting harmonic balance method. Setting minimal mass and vibration accelerated speed of total components as optimization objective function,the teeth number,module,helical angle as optimization design variable,the gear strength,transmission ratio and contact ratio as constraint condition,the vibration optimization modal of multi-gear transmission system is built. The best design variables are solved by compiling mixed- discrete optimization codes based on branch and bound algorithm. Then,the vibration response is recalculated,the results point out that the mean vibration acceleration and total mass are decreased by 33. 5 percent and 12. 96 percent respectively,which indicates the remarkable effect of vibration optimization.