Autonomous excavation method for electric shovel robots based on embodied perception

To enhance the automation level of electric shovels and promote the transition of open-pit mining from “manned shovel + unmanned haul truck” to “unmanned shovel + unmanned haul truck,” this paper proposes an embodied-perception–driven autonomous excavation method for electric-shovel robots based on a digital twin system. A Transformer model is developed using spatiotemporal consistency and deformable attention mechanisms to recognize surrounding targets from multi-view images and perform 3D environmental reconstruction, while LiDAR point clouds are integrated to compensate for image depth and improve reconstruction accuracy. A parallel hybrid neural network is then constructed to model the relationships among hoist-motor current, crowd-motor current, and excavation resistance, enabling real-time resistance perception during excavation. A digital twin system integrating modeling, embodied perception, autonomous excavation, and continual learning is established, within which a deep reinforcement learning (DQN)-based excavation policy is trained through virtual excavation simulations. To ensure generalization and transferability, a goal-driven DQN model is introduced. Furthermore, to address efficiency degradation when deploying the virtually trained DQN model to real machines, a force-sensing control strategy is proposed. This strategy continuously optimizes control commands based on real-time excavation resistance, thereby improving excavation efficiency. Experiments show that the 3D reconstruction error in object position and orientation is 5.3% and 1.1%, respectively, while the root-mean-square error in predicting tangential and normal excavation forces is 1.9% and 1.2%. In the digital twin environment, the DQN-based autonomous excavation reduces energy consumption by approximately 10% compared with constant-speed excavation, and the trained DQN model can be successfully transferred to a physical electric-shovel prototype to achieve the target bucket-fill rate. The proposed force-sensing control strategy further improves excavation performance by about 5% on top of the DQN model. These results demonstrate that the embodied-perception electric-shovel robot can effectively sense its surroundings, learn through virtual interaction within the digital twin system, achieve autonomous excavation, and continually improve its excavation performance in real scenarios.