For the problem that it is difficult to extract fault features of vibration signals of transmission shaft system and the actual situation that it is difficult to obtain a large number of fault samples in fault diagnosis，a fault diagnosis method for transmission shafting based on variational mode decomposition（VMD）sand particle swarm optimization support vector machine （PSO-SVM）is proposed. Firstly, the vibration signal of transmission shaft system is subjected to VMD decomposition, and intrinsic mode function IMF is obtained. Then, the energy value of IMF and the corresponding energy entropy are calculated. Finally, Particle swarm optimization（PSO） is used to optimize the parameters of support vector machine（SVM）, and the energy value and energy entropy of normalized IMF are input into the PSO-SVM to identify the working state and fault type of transmission shaft system. The experimental results show that the accuracy of the method is 94.44%, and it can diagnose the fault of transmission shaft system accurately and effectively.