forked from openjournals/joss-papers
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy path10.21105.joss.00060.jats
255 lines (255 loc) · 11.9 KB
/
10.21105.joss.00060.jats
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.2 20190208//EN"
"JATS-publishing1.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" dtd-version="1.2" article-type="other">
<front>
<journal-meta>
<journal-id></journal-id>
<journal-title-group>
<journal-title>Journal of Open Source Software</journal-title>
<abbrev-journal-title>JOSS</abbrev-journal-title>
</journal-title-group>
<issn publication-format="electronic">2475-9066</issn>
<publisher>
<publisher-name>Open Journals</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">60</article-id>
<article-id pub-id-type="doi">10.21105/joss.00060</article-id>
<title-group>
<article-title>hebbRNN: A Reward-Modulated Hebbian Learning Rule for
Recurrent Neural Networks</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author">
<contrib-id contrib-id-type="orcid">0000-0002-5179-3181</contrib-id>
<string-name>Jonathan A Michaels</string-name>
<xref ref-type="aff" rid="aff-1"/>
</contrib>
<contrib contrib-type="author">
<contrib-id contrib-id-type="orcid">0000-0001-6593-2800</contrib-id>
<string-name>Hansjörg Scherberger</string-name>
<xref ref-type="aff" rid="aff-1"/>
<xref ref-type="aff" rid="aff-2"/>
</contrib>
<aff id="aff-1">
<institution-wrap>
<institution>German Primate Center, Göttingen, Germany</institution>
</institution-wrap>
</aff>
<aff id="aff-2">
<institution-wrap>
<institution>Biology Department, University of Göttingen,
Germany</institution>
</institution-wrap>
</aff>
</contrib-group>
<pub-date date-type="pub" publication-format="electronic" iso-8601-date="2016-08-22">
<day>22</day>
<month>8</month>
<year>2016</year>
</pub-date>
<volume>1</volume>
<issue>5</issue>
<fpage>60</fpage>
<permissions>
<copyright-statement>Authors of papers retain copyright and release the
work under a Creative Commons Attribution 4.0 International License (CC
BY 4.0)</copyright-statement>
<copyright-year>2021</copyright-year>
<copyright-holder>The article authors</copyright-holder>
<license license-type="open-access" xlink:href="https://creativecommons.org/licenses/by/4.0/">
<license-p>Authors of papers retain copyright and release the work under
a Creative Commons Attribution 4.0 International License (CC BY
4.0)</license-p>
</license>
</permissions>
<kwd-group kwd-group-type="author">
<kwd>learning</kwd>
<kwd>plasticity</kwd>
<kwd>neural network</kwd>
<kwd>Hebbian</kwd>
<kwd>RNN</kwd>
</kwd-group>
</article-meta>
</front>
<body>
<sec id="summary">
<title>Summary</title>
<p>How does our brain learn to produce the large, impressive, and
flexible array of motor behaviors we possess? In recent years, there
has been renewed interest in modeling complex human behaviors such as
memory and motor skills using neural networks
(<xref alt="Carnevale et al., 2015" rid="ref-CarnevaleU003A2015jk" ref-type="bibr">Carnevale
et al., 2015</xref>;
<xref alt="Hennequin et al., 2014" rid="ref-HennequinU003A2014jh" ref-type="bibr">Hennequin
et al., 2014</xref>;
<xref alt="Laje et al., 2013" rid="ref-LajeU003A2013bd" ref-type="bibr">Laje
et al., 2013</xref>;
<xref alt="Rajan et al., 2016" rid="ref-RajanU003A2016cp" ref-type="bibr">Rajan
et al., 2016</xref>;
<xref alt="Sussillo et al., 2015" rid="ref-SussilloU003A2015kp" ref-type="bibr">Sussillo
et al., 2015</xref>). However, training these networks to produce
meaningful behavior has proven difficult. Furthermore, the most common
methods are generally not biologically-plausible and rely on
information not local to the synapses of individual neurons as well as
instantaneous reward signals
(<xref alt="Martens & Sutskever, 2011" rid="ref-MartensU003A2011vh" ref-type="bibr">Martens
& Sutskever, 2011</xref>;
<xref alt="Song et al., 2016" rid="ref-SongU003A2016fja" ref-type="bibr">Song
et al., 2016</xref>;
<xref alt="Sussillo & Abbott, 2009" rid="ref-SussilloU003A2009gh" ref-type="bibr">Sussillo
& Abbott, 2009</xref>).</p>
<p>The current package is a Matlab implementation of a
biologically-plausible training rule for recurrent neural networks
using a delayed and sparse reward signal
(<xref alt="Miconi, 2016" rid="ref-MiconiU003A2016dja" ref-type="bibr">Miconi,
2016</xref>). On individual trials, input is perturbed randomly at the
synapses of individual neurons and these potential weight changes are
accumulated in a Hebbian manner (multiplying pre- and post-synaptic
weights) in an eligibility trace. At the end of each trial, a reward
signal is determined based on the overall performance of the network
in achieving the desired goal, and this reward is compared to the
expected reward. The difference between the observed and expected
reward is used in combination with the eligibility trace to strengthen
or weaken corresponding synapses within the network, leading to proper
network performance over time.</p>
</sec>
</body>
<back>
<ref-list>
<ref-list>
<ref id="ref-CarnevaleU003A2015jk">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Carnevale</surname><given-names>Federico</given-names></name>
<name><surname>Lafuente</surname><given-names>Victor de</given-names></name>
<name><surname>Romo</surname><given-names>Ranulfo</given-names></name>
<name><surname>Barak</surname><given-names>Omri</given-names></name>
<name><surname>Parga</surname><given-names>Néstor</given-names></name>
</person-group>
<article-title>Dynamic Control of Response Criterion in Premotor Cortex during Perceptual Detection under Temporal Uncertainty.</article-title>
<source>Neuron</source>
<year iso-8601-date="2015-05">2015</year><month>05</month>
<volume>86</volume>
<issue>4</issue>
<pub-id pub-id-type="doi">10.1016/j.neuron.2015.04.014</pub-id>
</element-citation>
</ref>
<ref id="ref-HennequinU003A2014jh">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Hennequin</surname><given-names>Guillaume</given-names></name>
<name><surname>Vogels</surname><given-names>Tim P</given-names></name>
<name><surname>Gerstner</surname><given-names>Wulfram</given-names></name>
</person-group>
<article-title>Optimal control of transient dynamics in balanced networks supports generation of complex movements.</article-title>
<source>Neuron</source>
<year iso-8601-date="2014-06">2014</year><month>06</month>
<volume>82</volume>
<issue>6</issue>
<pub-id pub-id-type="doi">10.1016/j.neuron.2014.04.045</pub-id>
</element-citation>
</ref>
<ref id="ref-LajeU003A2013bd">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Laje</surname><given-names>Rodrigo</given-names></name>
<name><surname>Buonomano</surname><given-names>Dean V</given-names></name>
<name><surname>Buonomano</surname><given-names>Dean V</given-names></name>
</person-group>
<article-title>Robust timing and motor patterns by taming chaos in recurrent neural networks.</article-title>
<source>Nature Neuroscience</source>
<year iso-8601-date="2013-07">2013</year><month>07</month>
<volume>16</volume>
<issue>7</issue>
<pub-id pub-id-type="doi">10.1038/nn.3405</pub-id>
</element-citation>
</ref>
<ref id="ref-MartensU003A2011vh">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Martens</surname><given-names>J</given-names></name>
<name><surname>Sutskever</surname><given-names>I</given-names></name>
</person-group>
<article-title>Learning recurrent neural networks with hessian-free optimization</article-title>
<source>Proceedings of the 28th International Conference on Machine Learning</source>
<year iso-8601-date="2011">2011</year>
</element-citation>
</ref>
<ref id="ref-MiconiU003A2016dja">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Miconi</surname><given-names>Thomas</given-names></name>
</person-group>
<article-title>Flexible decision-making in recurrent neural networks trained with a biologically plausible rule</article-title>
<source>bioRxiv</source>
<year iso-8601-date="2016-07">2016</year><month>07</month>
<pub-id pub-id-type="doi">10.1101/057729</pub-id>
</element-citation>
</ref>
<ref id="ref-RajanU003A2016cp">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Rajan</surname><given-names>Kanaka</given-names></name>
<name><surname>Harvey</surname><given-names>Christopher D</given-names></name>
<name><surname>Tank</surname><given-names>David W</given-names></name>
</person-group>
<article-title>Recurrent Network Models of Sequence Generation and Memory.</article-title>
<source>Neuron</source>
<year iso-8601-date="2016-03">2016</year><month>03</month>
<volume>90</volume>
<issue>1</issue>
<pub-id pub-id-type="doi">10.1016/j.neuron.2016.02.009</pub-id>
</element-citation>
</ref>
<ref id="ref-SongU003A2016fja">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Song</surname><given-names>H Francis</given-names></name>
<name><surname>Yang</surname><given-names>Guangyu R</given-names></name>
<name><surname>Wang</surname><given-names>Xiao-Jing</given-names></name>
</person-group>
<article-title>Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework.</article-title>
<source>PLoS computational biology</source>
<year iso-8601-date="2016-02">2016</year><month>02</month>
<volume>12</volume>
<issue>2</issue>
<pub-id pub-id-type="doi">10.1371/journal.pcbi.1004792</pub-id>
</element-citation>
</ref>
<ref id="ref-SussilloU003A2015kp">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Sussillo</surname><given-names>David</given-names></name>
<name><surname>Churchland</surname><given-names>Mark M</given-names></name>
<name><surname>Kaufman</surname><given-names>Matthew T</given-names></name>
<name><surname>Shenoy</surname><given-names>Krishna V</given-names></name>
</person-group>
<article-title>A neural network that finds a naturalistic solution for the production of muscle activity.</article-title>
<source>Nature Neuroscience</source>
<year iso-8601-date="2015-07">2015</year><month>07</month>
<volume>18</volume>
<issue>7</issue>
<pub-id pub-id-type="doi">10.1038/nn.4042</pub-id>
</element-citation>
</ref>
<ref id="ref-SussilloU003A2009gh">
<element-citation publication-type="article-journal">
<person-group person-group-type="author">
<name><surname>Sussillo</surname><given-names>David</given-names></name>
<name><surname>Abbott</surname><given-names>L F</given-names></name>
</person-group>
<article-title>Generating coherent patterns of activity from chaotic neural networks.</article-title>
<source>Neuron</source>
<year iso-8601-date="2009-08">2009</year><month>08</month>
<volume>63</volume>
<issue>4</issue>
<pub-id pub-id-type="doi">10.1016/j.neuron.2009.07.018</pub-id>
</element-citation>
</ref>
</ref-list>
</ref-list>
</back>
</article>