Psych 16N Amines and Affect Professor Knutson June 1, 2009

Temporal subdivisions and their role in augmenting temporal precision and aggregate quantity of dopamine release in monkeys
David P. Havens1
Dopamine is known to be involved in predicting reward, but how can this system be enhanced !n monkeys, a repeated stimulus indicating a reward allows for greater firing of the dopamine neurons in the ventral tegmental area than a single stimulus indicating the same reward in the same amount of time. "y dividing up the time between stimulus onset and reward, the monkeys also e#hibited greater awareness of when the reward would come, measured by when they licked their lips in anticipation of the $uicy $uice%. These subdivisions have psychological implications as well & by counting down the seconds or days to an event, we increase the dopamine activity and our awareness of the event. 'lso, this information is practical in neuroeconomics( by reminding people of an upcoming event, you can increase their en)oyment, which can be useful in game planning and gambling in order to make more money. !ntroduction Advanced and rapid reward prediction systems help humans ma e decisions and optimi!e "ehavior when confronted with options of differin# value and certainty$ %ecisions and reward predictions ran#e #reatly in comple&ity, from Pavlov's do#s anticipatin# food to economic decisions that pro(ect years into the future$ )oth the prefrontal corte& and dopamine systems have "een implicated in helpin# ma e these decisions *1+, "ut as a reward #ets farther into the future or less intense, the dopamine system has a
1

1uic 2ime3 and a decompressor are needed to see this picture$

,nder#raduate, -tanford ,niversity$ .ontact/ dhavens0stanford$edu

4i#ure 1, from Professor -am 5c.lure, 6ecture 21 of Psych 70

si#nificantly lesser effect on the decision *7+$ 8n this respect, dopamine release detects "oth the timin# and ma#nitude of anticipated reward, nown as the scalar property of timin# and 9e"er's law for stimulus intensity, respectively$ 8t also differentiates "etween primary :such as (uice; and secondary rewards :such as money; *2+$ -ettin# aside thou#hts a"out lon#<term, calculated and thou#ht out decisions, let us focus on the more natural, impulsive, decisions, which can "e lar#ely dictated "y the dopamine reward prediction system$ 9e have seen that an unpredicta"le reward inspires a release of dopamine neurons, and when we si#nal this reward with a conditioned stimulus :.-;, the dopamine will release after the stimulus in a sort of anticipatory prediction of the reward to come, and not release at the reward itself$ 8f the anticipated reward is a"sent, however, after it has "een predicted with dopamine, there will "e a dearth of dopamine when the reward should have come, and the "rain will incorporate this uncertainty into its learnin# :4i#ure 1;$ 5any studies have "een done dealin# with this short<term reward prediction with respect to dopamine neurons$ As the reward #ets farther away from the .-, there is less of a release after the .- and a sli#ht release after the reward *=+$ 9hen uncertainty of the reward is increased to p > $7, the effect of the stimulus is almost nothin#, and the reward acts almost the same as an unpredicta"le reward *1+$ %elayin# the stimulus also creates a chan#e in the dopamine value system *?+$ 8n my research, 8 see to e&amine the effects of a repeated stimulus on dopamine release$

-2-

Proposal 9hen studyin# temporal discountin#, 4iorillo et al$ delivered a stimulus and waited 2<16 seconds "efore deliverin# a response$ 2he lon#er they waited, the less dopamine fired with the .- :scalar law;$ Also, the mon eys were not entirely precise with their timin#, often "e#innin# to salivate much "efore the actual stimulus :even thou#h they had "een fully trained$ 2his mi#ht "e due to a lac of necessary precision #iven the tas ; *=+$ 9hat happens to precision, thou#h, if instead of one li#ht "ul" flash leadin# up to the reward there were five e@ually spaced li#ht "ul" flashesA 9ould precision increaseA 2he mon ey mi#ht #et e&cited at the start of the stimulus, mi#ht "e increasin#ly more e&cited with each part of the stimulus, or mi#ht see the stimulus as a whole and not anticipate the reward until after the fifth flash$ 9ould there still "e any release after the reward itselfA Bopefully findin# the answer to these @uestions helps us understand the dopamine reward prediction system a little "etter$ *ethods Conditioning/ 2wo mon eys, either rhesus or maca@ue, will "e trained to three different stimuli/ a $7 second "eep that indicates a reward :(uice; in ten seconds, a concentrated $7 second li#ht flash that indicates a reward in ten seconds, and five li#ht flashes in two second intervals that indicate a reward ten seconds after the first flash, or two seconds after the fifth$ 8 am includin# "oth the "eep and the sin#le li#ht flash in order to see the effect that the am"i#uity of whether a second li#ht flash will follow the first will

-3-

have on dopamine firin#$ 2he mon eys will "e trained for at least five days and will have at least 700 e&posures to each stimulus to ma e sure it is fully conditioned$ 9e could follow the -wiss Animal Protection 6aw #uidelines or another international standard for how to study the mon eys$ 2he data will "e collected over a month, runnin# 100<200 tests a day$ Reward/ 2he (uice is sweet water :or may"e even (ust a drop of (uicy (uiceC; that will "e administered @uic ly in very small amounts :less than one m6 per dose; strai#ht to the mouth$ )etween trials, 8 will wait a random num"er of seconds "etween 20 and 60 "efore "e#innin# another stimulus, so as not to ma e the mon ey sic of (uice, "ut also to eep it from #ettin# on a re#ular schedule where the (uice is not si#naled "y the flashes "ut rather (ust at even 70 second intervals :for e&ample;$ Recording/ 2here are several via"le options for what to measure$ 8 could either measure the firin# rate usin# electrophysiolo#y in the ventral te#mental area or measure dopamine release usin# voltammetry in the nucleus accum"ens$ 8 have decided to measure the firin# rate usin# a sin#le unit recordin#Delectrophysiolo#y "ecause the temporal resolution is very #ood and it seems to "e the more standard thin# to do when measurin# activity of dopamine neurons$ 2o actually record these, 8 will "orrow 9olfram -chult!'s procedure outlined in *6+/ Animals had "een implanted for the precedin# study on reaction time tas s$ ,nder deep sodium pento"ar"ital anesthesia and aseptic condi< tions, cylinders for head fi&ation and a stereota&ically positioned, stain< less steel cham"er were fi&ed to the s ull to permit vertical access
-4-

with microelectrodes to the left su"stantia ni#ra$ 2he dura was left intact$ 2eflon<coated, multistranded, stainless steel wires were implanted into the e&tensor di#itorum communis and "iceps muscles of the arm and led su"cutaneously to the head$ A#<A#.l electrodes were implanted into the outer, upper, and lower canthi of the or"its$ All metal components, includin# connector plu#s for the muscle and perior"ital electrodes, were em"edded in several layers of dental cement and fi&ed to the s ull with sur#ical #rade stainless steel screws$ Ene wee after implantation, ani< mals were anestheti!ed with pento"ar"ital and the area of su"stantia ni#ra was locali!ed "y ta in# lateral and coronal radio#raphs with a #uide cannula installed at a nown coordinate in reference to the im< planted steel cham"er$ 2he ventroposteromedial thalamus overlyin# the lateral su"stantia ni#ra was electrophysiolo#idcally e&plored for tri#em< inal input on the same occasion, and later occasionally in the wa in# animal$ :-chult! et al; 2he impulses were then measured usin# computers and platinum<plated tun#sten microelectrodes$ Fach neuron, once determined to "e dopaminer#ic, will "e recorded from 20 times$ Data Analysis: 2he neurons will "e compiled into one #raph, much li e the 4i#ure 1$ 2his can safely "e done as the dopamine neurons are all lar#ely homo#enous *=+$ 2he data set of the unpredicted reward will "e used as a "aseline$ Independent Variables/ the type of stimulus :either one sound or one flash or five flashes; and whether or not a reward is delivered$ 2rial A 2ime 0 H 2 H ? H 6 H
-5-

G H

10 Juice

Table 1: Independent Variables

) . % F 4 I

)eep 4lash 4lash )eep 4lash 4lash

H H 4lash H H 4lash

H H 4lash H H 4lash

H H 4lash H H 4lash

H H 4lash H H 4lash

Juice Juice Juice No Juice No Juice No Juice

Fixed Variables: 2ime of reward with respect to stimulus onset, type of (uice, intensity of flash, ma#nitude of (uiceDreward$ Dependent Variables: 2he firin# rate of the dopamine neurons$ +esults ,invented & numbers appro#imated from -./0

-6-

Scenario 1. 2he dopamine neurons measured had a tonic firin# rate of two impulses per second$ 9hen the "eep si#naled a reward in ten seconds, the dopamine rate increased to ei#ht for a short period of time "efore returnin# to the tonic rate$ 9hen the (uice came, the neurons fired almost as thou#ht it were an une&pected reward, (umpin# up to 1? impulses per second$ 2he flash, which was almost li e the "eep, fell in second two when the second flash did not come, then returned to tonic$ 2he (uice came even close to "ein# an une&pected reward$ 9ith multiple flashes, it "e#an (ust li e a normal flash, "ut when the second,
4i#ure 2 J avera#ed from all recordin#s from all neurons

affirmin# flash came, it inspired another anticipatory release of dopamine "efore "ein# (ust a"ove tonic until the ei#hth second, when it (umped way up to ten impulsesDsecond$ 9hen the (uice came, there were only five impulses per second, which ma es trial % fall "etween trial . and the predicta"le reward trial from 4i#ure 1$ 9hen the (uice is withheld, there is an a"sence of dopamine, (ust li e in the e&ample with only two seconds "etween .- and reward :fi#ure 1;$ 2here is a #reater a"sence of firin# for the multiple flash

-7-

system$

Table 2: Dependent Variables (numbers are the avera e !" all trials#

2ria l A ) . % F 4 I

2im e 0 H )ee p 4las h 4las h )ee p 4las h 4las h

2 H H H 4lash H H 4lash

? H H H 4lash H H 4lash

6 H H H 4lash H H 4lash

G H H H 4las h H H 4las h

10 Juice Juice Juice Juice No Juice No Juice No Juice

.:impulseDs ec; 4irin# rate 2 :tonic rate; G, app&$ tonic G, 1, 2, 2, 2 G, 6, =, =, 10 G, app&$ tonic G, 1, 2, 2, 2 G, 6, =, =, 10

Juice :impulseDs ec; 4irin# rate 16 1? 17 7 1 1 0

)ehaviorally, the mon ey's lic their lips to show anticipation$ 8n trials ), ., F, and 4, the mon ey's start to lic their lips around second 7$ 8n trials % and I, they wait until the Gth second to lic their lips in anticipation$ 2his shows #reater temporal precision$ Scenario 2. 8t mi#ht "e that there is no chan#e in the five flash situation, "ut this doesn't seem li ely or as interestin#, so 8 have not considered it$ Also, ) and . could "e e&actly the same, 8 only included ) "ecause 8 fi#ured . mi#ht yield interestin# results "ecause % is a

-8-

continuation of it, so the mon ey mi#ht e&pect % "ut only #et . and have a sort of mini<prediction<error<correction to ma e as the tas contin!es$ As far as lip lic in# #oes, it mi#ht "e that they start lic in# sooner when there are multiple flashes :in anticipation; instead of waitin# until the last flash$ 8t also mi#ht "e that lip lic in#, to "e most effective, starts five seconds early$ A#ain, this is less interestin#$ 8 "asically am ma in# a hypothesis in scenario 1 and then pretendin# that it has come true in the discussion, which follows$ Discussion -o how does this relate to the real worldA 2he multiple flash system mi#ht "e analo#ous to the psycholo#y of countin# down$ 9e do it all the time to add enthusiasm to life J countin# down the end of a #ame, countin# down the New Kear$ 8f one (ust said Lten seconds leftM and then waited for the end to come, it mi#ht "e more of an une&pected outcome than with the continued thou#ht and reassessment of where one is in temporal relation to the #oal$ As discovered in *=+, the dopamine system is not strictly temporally precise :althou#h this idea needs to "e tested with e&periments where precision is necessary;$ Note that the multiple flash system also :at least in this invention; has a #reater total num"er of impulses in the same amount of time, leadin# me to conclude that countin# down to a reward ma&imi!es dopamine :and in a way, the rewardin#<ness of the reward;$ 2his ma es sense in life, as people often count down the last five :or fifty; days of wor "efore a vacation, constantly re<updatin# their temporal precision, and when they re<position themselves temporally they often feel happy, perchance a

-$-

"urst of anticipatory dopamine$ 8f we stic strictly to the analo#y, too, we see that althou#h they were happy when thin in# a"out the vacation, they are less happy when it finally #ets there than someone who lost trac of time who is surprised "y the arrival of the reward$ .ountin# down, however, still ma&imi!es dopamine over time, even if it is more spread out$ Iranted, for somethin# as "i# as a vacation, many other "rain systems must come into consideration$ Also, this information has relevance to #am"lin# and #ame desi#n$ 2o now that people can #et the most rush if they are constantly reminded of the fortune they are a"out to win is valua"le nowled#e, especially when considerin# too that once they win, they actually e&perience less dopamine action than when they thou#ht they were #oin# to win :if they truly thou#ht they were a"out to winD#et a reward;$ As a #ame show host, then, you should always lead the contestant on to "elieve he or she can win in order to optimi!e dopamine activity$ Fven if they don't win, the overall activity would have "een more dopaminer#ic and the contestant more li ely to come "ac for a second round :ie spend more money at your casino;$ Also, the countin# downDmultiple flash scenario pro"a"ly :hypothesis; leads to a more accurate value prediction, as one is constantly reassessin# the situation$ All in all, repeated si#nalin# enhances the anticipation of a reward "ut diminishes the reward's impact upon reception when compared to a sin#le stimulus "e#un at the same time$ 'cknowledgements

- 1% -

2han s to )rian Knutson for feed"ac on the initial research proposal$ 1orks 2ited *1+ -hult!, 9olfram$ )ehavioral 2heories and the Neurophysiolo#y of Neward$ Ann!al Re"iew o# $sychology. 34/ GO<117 :2006; *2+ %PArdenne K, 5c.lure -5, Nystrom 6F, .ohen J%$ )E6% responses reflectin# dopaminer#ic si#nals in the human ventral te#mental area$ Science$ .15,36740( 126?<O$ :200G; *=+ 4iorillo .%, Newsome 92, -chult! 9$ 2he 2emporal Precision of Neward prediction in dopamine neurons$ %at %e!rosci. F<pu"$ :200G; *?+ Ko"ayashi -, -chult! 9$ 8nfluence of Neward %elays on Nesponses of %opamine Neurons$ & %e!rosci$ 86,.10/ OG=O<?6$ :200G; *7+ 5c.lure, -am$ L%ecision Neuroscience$M 6ecture at -tanford ,niversity$ April 21, 2009 *6+ 9olfram -chult!, Paul Apicella, and 2omas 6(un#"er#"$ Nesponses of 5on ey %opamine Neurons to Neward and .onditioned -timuli durin# -uccessive -teps of 6earnin# a %elayed Nesponse 2as $ 'he &o!rnal o# %e!roscience, 1.,.0( 900<91= :199=;

- 11 -