ACE (Actor-Critic-Explorer) paradigm for reinforcement learning in basal ganglia: Highlighting the role of subthalamic and pallidal nuclei

We present a comprehensive model of basal ganglia in which the three important reinforcement learning components-Actor, Critic and Explorer (ACE),-are represented and their anatomical substrates are identified. Particularly, we identify the subthalamic-nucleus and globus pallidus externa (STN-GPe) loop as the Explorer, and argue that complex activity of STN and GPe neurons, found in experimental studies, provides the stochastic drive necessary for exploration. Simulations involving a two-link arm model show task-dependent variations in complexity of STN-GPe activity when the ACE network is trained to perform simple reaching movements. Complexity and average levels of STN-GPe activity are observed to be higher before training than in post-training conditions. Further, in order to simulate Parkinsonian conditions, when dopamine levels in substantia nigra portion of the model are reduced, the arm displayed, as a primary change, small amplitude movements, which on persistent network training, amplified to large amplitude unregulated movements reminiscent of Parkinsonian tremor. © 2010 Elsevier B.V.