Abstract: In order to study the main controlling factors of borehole deformation and instability in deep coal seam, firstly, the mechanism of borehole deformation and instability is analyzed. Secondly, a three-dimensional numerical model is established to study the influence of borehole buried depth and pressure coefficient on borehole stability. Finally, the in-situ stress is determined as the main controlling factor of borehole deformation and instability. The results show that:① with the increase of borehole depth, the maximum vertical displacement, maximum horizontal displacement and maximum vertical stress of surrounding rock increase, and the burial depth increases by one time, the maximum displacement increases by 6.11 times, the maximum vertical pressure increases by one time, the distance between the maximum vertical stress position and the borehole surface increases, but the vertical stress concentration factor is basically unchanged. ② With the increase of the lateral pressure coefficient, the maximum vertical displacement and the maximum horizontal displacement of the surrounding rock of the borehole increase, and the maximum vertical displacement increases faster, and the maximum displacement of the borehole becomes 3.29 times of the original when the lateral pressure coefficient is doubled; the maximum vertical stress of the borehole increases slowly with it. ③ Hole diameter and gas pressure have little influence on borehole stability. The maximum vertical displacement, maximum horizontal displacement and maximum vertical stress increase slowly with the increase of pore diameter or gas pressure. ④ When the casing is used to support the borehole, the borehole deformation is controlled to a certain extent, the maximum vertical displacement and the maximum horizontal displacement are significantly reduced, the distance between the maximum vertical stress and the borehole and the stress concentration factor are reduced, and the effect of PP casing is the best. ⑤ The in-situ stress determined by buried depth and lateral pressure coefficient(geological structure) is the source of borehole instability and the main controlling factor affecting the stability of extraction borehole. The research results can provide a theoretical basis for mine gas control.