Abstract: The room-and-pillar mined-out area has the characteristics of small recovery rate and large coal pillar, and its overburden movement law and surface deformation law are quite different from those of long-wall mined-out area. In the case of coal pillar failure, the surface will produce large deformation, threatening the safety of surface structures. In order to study the deformation and development mechanism of overlying rock and unconsolidated formation in shallow and gently tilted room-and-pillar mined-out area, based on PIV image processing technology and modular assembly idea, a physical model was established in the laboratory to carry out physical simulation experiments, inversion of the deformation and development process of overlying rock and unconsolidated formation in room-and-pillar mined-out area, and analysis of the deformation mechanism of overlying rock and unconsolidated formation under the condition of room-and-pillar mining and coal pillar instability. The research results show that: ① the overburden deformation of shallow buried and gently mined goaf can be divided into three areas: “banded” deformation area, “trapezoidal” deformation area and “inverted funnel” deformation area. The deformation in the unconsolidated formation is “trapezoidal”. ② Under the instability condition of coal pillar, the overburden deformation law remains unchanged, and the deformation characteristics of unconsolidated formation develop from “trapezoidal” to “funnel”. With the increase of unstable coal pillar, the deformation of unconsolidated formation increases, and the “funnel” deformation area gradually expands from the center to both ends. ③ After the end of room-and-pillar mining, the stress of the goaf roof can be simplified to simply supported beams bearing uniform load. In the case of continuous instability of coal pillar, the static load of overlying rock and unconsolidated formation will lead to the goaf roof reaching the ultimate equilibrium state. The failure mode of the roof is tensile breaking, and cracks are generated under the action of tensile stress, but the roof is not completely broken. The goaf roof forms a cantilever beam structure with cracks. The instability of coal pillar in shallow and gently tilting room-and-pillar mined-out area has a strong influence on the surface deformation, and the deformation of unconsolidated formation and overlying rock are in the same order of magnitude.