The variational mode decomposition（ VMD） is a novel self-adaptive decomposition method without shortcomings of traditional adaptive decomposition methods,such as mode mixing and end effects. The VMD has the advantages of strong anti-noise robustness and high efficiency. The generalized Warblet transform（ GWT） is suitable for analyzing modulated signal with periodic or nonperiodic change of instantaneous frequency. Considering the advantages of VMD in decomposing signal into sub-signals,a new method is proposed to detect gear fault based on GWT and VMD in this work. The VMD can be considered as the signal preprocessing prior to GWT. The results of the experimental analysis have demonstrated the effectiveness of the proposed method.