Solving Sparse Reward Tasks Using Self-Balancing Exploration and Exploitation

Abstract

A core challenge in applying deep reinforcement learning (DRL) to real-world tasks is the sparse reward problem, and shaping reward has been one effective method to solve it. However, due to the enormous state space and sparse rewards in the real world, a large number of useless samples may be generated, leading to reduced sample efficiency and potential local optima. To address this issue, this study proposes a self-balancing method of exploration and development to solve the issue of sparse rewards. Firstly, we shape the reward function according to the evaluated progress, to guide the agent's learning of high-reward samples. Secondly, we construct a dual-trajectory exploration network, which provides intrinsic rewards based on the novelty of states and the trajectory difference of sibling agents to encourage the agent to explore and adjust the balance between exploration and exploitation. This method effectively prevents the generation of a large amount of useless training data during the interaction between the agent and the environment, resolves local optimal dilemmas through state novelty, and adjusts the strategy in a timely manner to solve sparse reward tasks. Our method outperforms basic reinforcement learning (RL) and curiosity-driven incentives in these experimental tasks. The self-balancing exploration and exploitation approach in our research provides a new perspective and effective solution for addressing the problem of sparse rewards, thereby advancing the application of DRL in real-world problems and achieving greater success.