For Dual-tree complex wavelet transform decomposition of the number of layers need to be determined prior to and the sub-band after the occurrence of frequency aliasing, an improved dual tree complex wavelet transform is proposed to deal with gearbox composite fault feature extraction. Firstly, the decomposition layer number and the effective sub-band of the dual-tree complex wavelet transform are adaptively determined. The obtained sub-bands are subjected to frequency removal, the sub-band FFT is used to determine the main frequency and its notch filtering is performed to ensure that the frequency aliasing is eliminated, so that each sub-band only contains a unique characteristic frequency. Then using the proposed method and the methods such as Variational Mode Decomposition, the noise-containing simulation signals are decomposed and compared to verify the feasibility of the proposed method. The proposed method is applied to the vibration signal of the gearbox composite fault and successfully extracted the gear spalling and bearing outer ring fault. The proposed method provides a new idea for the extraction of complex fault features of gearboxes.