Structure Learning in Motor Control:A Deep Reinforcement Learning Model

Motor adaptation displays a structure-learning effect: adaptation to a new perturbation occurs more quickly when the subject has prior exposure to perturbations with related structure. Although this `learning-to-learn' effect is well documented, its underlying computational mechanisms are poorly understood. We present a new model of motor structure learning, approaching it from the point of view of deep reinforcement learning. Previous work outside of motor control has shown how recurrent neural networks can account for learning-to-learn effects. We leverage this insight to address motor learning, by importing it into the setting of model-based reinforcement learning. We apply the resulting processing architecture to empirical findings from a landmark study of structure learning in target-directed reaching (Braun et al., 2009), and discuss its implications for a wider range of learning-to-learn phenomena.Comment: 39th Annual Meeting of the Cognitive Science Society, to appea

Neural correlates of cognitive dissonance and choice-induced preference change

According to many modern economic theories, actions simply reflect an individual's preferences, whereas a psychological phenomenon called “cognitive dissonance” claims that actions can also create preference. Cognitive dissonance theory states that after making a difficult choice between two equally preferred items, the act of rejecting a favorite item induces an uncomfortable feeling (cognitive dissonance), which in turn motivates individuals to change their preferences to match their prior decision (i.e., reducing preference for rejected items). Recently, however, Chen and Risen [Chen K, Risen J (2010) J Pers Soc Psychol 99:573–594] pointed out a serious methodological problem, which casts a doubt on the very existence of this choice-induced preference change as studied over the past 50 y. Here, using a proper control condition and two measures of preferences (self-report and brain activity), we found that the mere act of making a choice can change self-report preference as well as its neural representation (i.e., striatum activity), thus providing strong evidence for choice-induced preference change. Furthermore, our data indicate that the anterior cingulate cortex and dorsolateral prefrontal cortex tracked the degree of cognitive dissonance on a trial-by-trial basis. Our findings provide important insights into the neural basis of how actions can alter an individual's preferences

Higher media multi-tasking activity is associated with smaller gray-matter density in the anterior cingulate cortex

Media multitasking, or the concurrent consumption of multiple media forms, is increasingly prevalent in today’s society and has been associated with negative psychosocial and cognitive impacts. Individuals who engage in heavier media-multitasking are found to perform worse on cognitive control tasks and exhibit more socio-emotional difficulties. However, the neural processes associated with media multi-tasking remain unexplored. The present study investigated relationships between media multitasking activity and brain structure. Research has demonstrated that brain structure can be altered upon prolonged exposure to novel environments and experience. Thus, we expected differential engagements in media multitasking to correlate with brain structure variability. This was confirmed via Voxel-Based Morphometry (VBM) analyses: Individuals with higher Media Multitasking Index (MMI) scores had smaller gray matter density in the anterior cingulate cortex (ACC). Functional connectivity between this ACC region and the precuneus was negatively associated with MMI. Our findings suggest a possible structural correlate for the observed decreased cognitive control performance and socio-emotional regulation in heavy media-multitaskers. While the cross-sectional nature of our study does not allow us to specify the direction of causality, our results brought to light novel associations between individual media multitasking behaviors and ACC structure differences

Evidence for surprise minimization over value maximization in choice behavior

Classical economic models are predicated on the idea that the ultimate aim of choice is to maximize utility or reward. In contrast, an alternative perspective highlights the fact that adaptive behavior requires agents' to model their environment and minimize surprise about the states they frequent. We propose that choice behavior can be more accurately accounted for by surprise minimization compared to reward or utility maximization alone. Minimizing surprise makes a prediction at variance with expected utility models; namely, that in addition to attaining valuable states, agents attempt to maximize the entropy over outcomes and thus 'keep their options open'. We tested this prediction using a simple binary choice paradigm and show that human decision-making is better explained by surprise minimization compared to utility maximization. Furthermore, we replicated this entropy-seeking behavior in a control task with no explicit utilities. These findings highlight a limitation of purely economic motivations in explaining choice behavior and instead emphasize the importance of belief-based motivations

Design for a Darwinian Brain: Part 1. Philosophy and Neuroscience

Physical symbol systems are needed for open-ended cognition. A good way to understand physical symbol systems is by comparison of thought to chemistry. Both have systematicity, productivity and compositionality. The state of the art in cognitive architectures for open-ended cognition is critically assessed. I conclude that a cognitive architecture that evolves symbol structures in the brain is a promising candidate to explain open-ended cognition. Part 2 of the paper presents such a cognitive architecture.Comment: Darwinian Neurodynamics. Submitted as a two part paper to Living Machines 2013 Natural History Museum, Londo

Under pressure: Response urgency modulates striatal and insula activity during decision-making under risk

When deciding whether to bet in situations that involve potential monetary loss or gain (mixed gambles), a subjective sense of pressure can influence the evaluation of the expected utility associated with each choice option. Here, we explored how gambling decisions, their psychophysiological and neural counterparts are modulated by an induced sense of urgency to respond. Urgency influenced decision times and evoked heart rate responses, interacting with the expected value of each gamble. Using functional MRI, we observed that this interaction was associated with changes in the activity of the striatum, a critical region for both reward and choice selection, and within the insula, a region implicated as the substrate of affective feelings arising from interoceptive signals which influence motivational behavior. Our findings bridge current psychophysiological and neurobiological models of value representation and action-programming, identifying the striatum and insular cortex as the key substrates of decision-making under risk and urgency