Abstract: In the process of applying the mining technology of the end-wall shearer to recover the coal resources retained in the end-wall, the reasonable design of the parameters of the supporting coal pillar is the premise of the safe and efficient end-wall mining. Based on the cusp catastrophe theory, the necessary and sufficient conditions for the instability of the supporting coal pillar in the end slope mining are established. The mechanical analysis model of mining roof-coal pillar was established. Based on the elastic foundation beam theory, the spatial evolution law of roof stress and deformation after local instability of coal pillar was studied, and the instability criterion of mining roof collapse was proposed, which was used as the failure criterion of supporting coal pillar under non-uniform load. The variation law of the limit caving distance of the roof and the length of the coal pillar instability section under different coal pillar widths was studied, and the design method of the supporting coal pillar parameters under non-uniform load was proposed. The results show that the necessary and sufficient condition of coal pillar instability is the bifurcation set Δ < 0, the instability criterion of the roof is that the maximum tensile stress σ_tmax is greater than the tensile strength σ_tl , the failure criterion of the coal pillar is that the length l_max of the instability section of the coal pillar is greater than the limit caving distance l_2max of the roof of the mining cave , the width of the coal pillar in the end-wall mining of an open-pit coal mine in Shanxi is designed to be 3.0 meters.