适用于强化学习惯性环境的分数阶改进OU噪声

An improved Ornstein-Uhlenbeck exploration noise based on fractional order calculus for reinforcement learning environments with momentum

摘要:

本文将DDPG算法中使用的OrnsteinUhlenbeck (OU) 噪声整数阶微分模型推广为分数阶OU噪声模型，使得噪声的产生不仅和前一步的噪声有关而且和前K步产生的噪声都有关联.通过在gym惯性环境下对比基于分数阶OU噪声的DDPG和TD3算法和原始的DDPG和TD3算法，我们发现基于分数阶微积分的OU噪声相比于原始的OU噪声能在更大范围内震荡，使用分数阶OU噪声的算法在惯性环境下具有更好的探索能力，收敛得更快.

Abstract:

In this paper, the integer-order Ornstein-Uhlenbeck (OU) noise model used in the deep deterministic policy gradient (DDPG) algorithm is extended to the fractionalorder OU noise model, and the generated noise is not only related to the noise of the previous step but also related to the noise generated in the previous K steps in the proposed model.The DDPG algorithm and twin delayed deep deterministic(TD3) algorithm using the fractional-order OU noise model were compared with the original DDPG algorithm and TD3 algorithm in the gym inertial environment. We found that, compared with the original OU noise, the fractional-order OU noise can oscillate in a wider range, and the algorithm using the fractional-order OU noise had better exploration ability and faster convergence in inertial environment.