Abstract: The top-coal caving technique is an effective method for extracting thick or ultra-thick coal seams, with numerical simulations commonly employed to study the caving and coal release processes. However, current discrete element-based numerical models are limited in terms of parameter determination, which affects computational accuracy and efficiency. Therefore, a novel mesh-free method, namely the Smoothed Particle Hydrodynamics (SPH), was adopted to develop a coal release model based on an elasto-plastic constitutive relationship. By leveraging the SPH method’s advantages in large deformation simulations and integrating a non-cohesive soil constitutive model with the Drucker-Prager yield criterion, the effects of various material parameters (e.g., friction angle, density) on the evolution of the coal-gangue interface was compared and analyzed. Additionally, the release and flow behavior of fragmented top-coal and gangue was simulated, and the model’s applicability in simulating coal-gangue interface evolution and particle flow was experimentally validated. Different coal release methods (mid-level and low-level caving) were analyzed to investigate the flow patterns of top-coal during support advancement. The results indicate that the non-cohesive model effectively simulates granular particle flow behavior, with the landslide simulation results aligning well with experimental data. The improved SPH algorithm addresses numerical noise issues, and the application of stress smoothing techniques reveals a “double-peak” stress distribution pattern. A smaller friction angle correlates with a greater downward displacement of the conical coal-gangue interface during release, while gangue density has no significant impact on the coal-gangue interface. Increased gangue thickness leads to a corresponding increase in coal release. The flow and velocity distribution patterns of top-coal under different caving methods align with the coal-gangue evolution mechanism, with low-level caving yielding a gangue content in top-coal of approximately 28.86%, around 16.89% lower than that of mid-level caving, indicating potential for further optimization. The SPH model developed requires only five material parameters, all of which can be obtained through standard geotechnical shear tests, enhancing the determinacy of the numerical model.