Reinforcement Learning in Noisy and Non-Markovian Domains

Mark Pendrith
School of Computer Science and Engineering, The University of New South Wales, Sydney 2052, Australia.


If reinforcement learning techniques are to be used for ``real world'' dynamic system control, the problems of noise and plant disturbance will have to be addressed. This study investigates the effects of noise/disturbance on five different RL algorithms: Watkins' 1-step Q-Learning; Barto, Sutton and Anderson's Adaptive Heuristic Critic; Sammut and Law's modern variant of Michie and Chambers' BOXES algorithm; and two new algorithms developed during the course of this study. Both these new algorithms are conceptually related to Q-Learning; both algorithms, called P-Trace and Q-Trace respectively, provide for substantially faster learning than straight Q-Learning overall, and for dramatically faster learning (by up to a factor of 200) in the special case of learning in a noisy environment for the dynamic system studied here (a pole-and-cart simulation).

As well as speeding learning, both the P-Trace and Q-Trace algorithms have been designed to preserve the ``convergence with probability 1'' formal properties of standard Q-Learning, i.e. that they be provably ``correct'' algorithms for Markovian domains for the same conditions that Q-Learning is guaranteed to be correct. We present both arguments and experimental evidence that ``actual return'' methods may prove to be both faster and more powerful in general than temporal difference methods. The potential performance improvements using actual return over pure temporal difference methods may turn out to be particularly important when learning is to occur in noisy or stochastic environments, and in the case where the domain is not well-modelled by Markovian processes.

Conference Home Page