NECO_a_00419.pdf (1.55 MB)
Solving the distal reward problem with rare correlations
journal contribution
posted on 2015-03-13, 13:46 authored by Andrea SoltoggioAndrea Soltoggio, Jochen SteilIn the course of trial-and-error learning, the results of actions, manifested as rewards or punishments, occur often seconds after the actions that caused them. How can a reward be associated with an earlier action when the neural activity that caused that action is no longer present in the network? This problem is referred to as the distal reward problem. A recent computational study proposes a solution using modulated plasticity with spiking neurons and argues that precise firing patterns in the millisecond range are essential for such a solution. In contrast, the study reported in this letter shows that it is the rarity of correlating neural activity, and not the spike timing, that allows the network to solve the distal reward problem.In this study, rare correlations are detected in a standard rate-based computational model by means of a thresholdaugmented Hebbian rule. The novel modulated plasticity rule allows a randomly connected network to learn in classical and instrumental conditioning scenarios with delayed rewards. The rarity of correlations is shown to be a pivotal factor in the learning and in handling various delays of the reward. This study additionally suggests the hypothesis that short-term synaptic plasticity may implement eligibility traces and thereby serve as a selectionmechanism in promoting candidate synapses for long-term storage.
Funding
This work was supported by the European Community’s Seventh Framework Programme FP7/2007-2013, Challenge 2 Cognitive Systems, Interaction, Robotics (Grant No. 248311—AMARSi).
History
School
- Science
Department
- Computer Science
Published in
Neural ComputationVolume
25Issue
4Pages
940 - 978Citation
SOLTOGGIO, A. and STEIL, J., 2013. Solving the distal reward problem with rare correlations. Neural Computation, 25 (4), pp. 940-978Publisher
© Massachusetts Institute of Technology PressVersion
- VoR (Version of Record)
Publisher statement
This work is made available according to the conditions of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) licence. Full details of this licence are available at: https://creativecommons.org/licenses/by-nc-nd/4.0/Publication date
2013Notes
This article is © Massachusetts Institute of Technology Press.ISSN
0899-7667eISSN
1530-888XPublisher version
Language
- en