Abstract: In order to maximize the benefit of coal mining in the open pit mine,based on the Strength Reduction Theory,a three-dimensional model was established by FLAC3D finite difference element software,the effects of the instability length and deep excavation slope angle of the coal compression zone were used to study the instability mode and stability of the composite slope and the optimum spatial shape of composite slope was determined.The results show that the failure mode and stability of composite slope are mainly controlled by the exposed length of the weak layer and the excavation angle of deep excavation.When the slope angle of deep excavation is more than 25°,with the increase of the exposure length of the weak layer,it obviously decreases first,and then approaching level development trend.When the exposed length of weak layer is more than 200 m, it is obviously reduced with the increase of deep excavation slope angle.The spatial effect of the horizontal mining and drainage method is applied to optimize the rear slope,and the angle of composite slope can be increased to 16°,and the angle of deep excavation increases to 25°,there 6.121 million tons of coal can be maximally extracted.