The chaotic motion and controlling of rotary machinery simplied of a compaction mechanism installed on the elastic support are studied. The results show that,there is a transformation from periodic motion to chaotic motion in period- doubling bifurcation route,then return to the periodic motion along with the increasing of angular velocity of the equivalent eccentric mass rotor. For the purpose of controlling chaos,a method to determine the design parameters of controller is proposed based on the edge of chaos. The main idea of the method is to take the parameters of controller as the system parameters,and the edge of chaos is determined,from which the design parameters of controller can be chosen directly. The method and its effectiveness are demonstrated by the design of PD controller and phase controller.