Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcement driving both action acquisition and reward maximization: A simulated robotic study

An important issue of recent neuroscientific research is to understand the functional role of the phasic release of dopamine in the striatum, and in particular its relation to reinforcement learning. The literature is split between two alternative hypotheses: one considers phasic dopamine as a reward prediction error similar to the computational TD-error, whose function is to guide an animal to maximize future rewards; the other holds that phasic dopamine is a sensory prediction error signal that lets the animal discover and acquire novel actions. In this paper we propose an original hypothesis that integrates these two contrasting positions: according to our view phasic dopamine represents a TD-like reinforcement prediction error learning signal determined by both unexpected changes in the environment (temporary, intrinsic reinforcements) and biological rewards (permanent, extrinsic reinforcements). Accordingly, dopamine plays the functional role of driving both the discovery and acquisition of novel actions and the maximization of future rewards. To validate our hypothesis we perform a series of experiments with a simulated robotic system that has to learn different skills in order to get rewards. We compare different versions of the system in which we vary the composition of the learning signal. The results show that only the system reinforced by both extrinsic and intrinsic reinforcements is able to reach high performance in sufficiently complex conditions

Cumulative learning through intrinsic reinforcements

Building artificial agents able to autonomously learn new skills and to easily adapt in different and complex environments is an important goal for robotics and machine learning. We propose that providing reinforcement learning artificial agents with a learning signal that resembles the charac- teristic of the phasic activations of dopaminergic neurons would be an advancement in the development of more autonomous and versatile systems. In particular, we suggest that the particular composition of such a signal, determined by both extrinsic and intrinsic reinforcements, would be suitable to improve the implementation of cumulative learning in artificial agents. To validate our hypothesis we performed experiments with a simulated robotic system that has to learn different skills to obtain extrinsic rewards. We compare different versions of the system varying the composition of the learning signal and we show that the only system able to reach high performance in the task is the one that implements the learning signal suggested by our hypothesis

A bio-inspired learning signal for the cumulative learning of different skills

Building artificial agents able to autonomously learn new skills and to easily adapt in different and complex environments is an important goal for robotics and machine learning. We propose that providing artificial agents with a learning signal that resembles the characteristic of the phasic activations of dopaminergic neurons would be an advancement in the development of more autonomous and versatile systems. In particular, we suggest that the particular composition of such a signal, determined both by intrinsic and extrinsic reinforcements, would be suitable to improve the implementation of cumulative learning. To validate our hypothesis we performed some experiments with a simulated robotic system that has to learn different skills to obtain rewards. We compared different versions of the system varying the composition of the learning signal and we show that only the system that implements our hypothesis is able to reach high performance in the task

Biological cumulative learning through intrinsic motivations: a simulated robotic study on development of visually-guided reaching

This work aims to model the ability of biological organisms to achieve cumulative learning, i.e. to learn increasingly more complex skills on the basis of simpler ones. In particular, we studied how a simulated kinematic robotic system composed of an arm and an eye can learn the ability to reach for an object on the basis of the ability to systematically look at the object, which, in our set-up, represented a prerequisite for the reaching task. We designed the system by following several biological constraints and investigated which kind of sub-task reinforcements might facilitate the development of the final skill. We found that the performance in the reaching task was optimized when the reinforcement signal included not only the extrinsic reinforcement provided by touching the object but also an intrinsic reinforcement given by the error in the prediction of fovea activation. We discuss how these results might explain biological data regarding the neural basis of action discovery and reinforcement earning, in particular with respect to the neuromodulator dopamine

La (mancata) tutela dei diritti fondamentali frav sfide del costituzionalismo (inter)nazionale e disordine delle fonti del diritto

This paper aims at providing an overview of the nearly seventy years of challenges which have been coming along with (inter)national constitutionalism since its beginnings. Most of all, it aims at drawing attention to their peculiar relevance with regard to both the ever growing disorder of (national and international) legal sources and the wide and larger uncertainty in the forms to protecting fundamental rights and preserving fundamental goods (§ 5). In particular, after a few remarks on its (failed) achievements (§ 2), two sorts of challenges of the (inter)national constitutionalism will be distinguished: (i) the (earlier) challenges which even before the conclusion of the second world war had marked its beginnings and the slowly progressive assessment of its ultimate principles (§ 3); and (ii) the (new) challenges which at the conclusion of the cold war, in the contest of a new geopolitical disorder, have started to undermining its hardly achieved results as well as the basic values of its legal and political project (§ 4)

Which is the best intrinsic motivation signal for learning multiple skills?

Humans and other biological agents are able to autonomously learn and cache different skills in the absence of any biological pressure or any assigned task. In this respect, Intrinsic Motivations (i.e., motivations not connected to reward-related stimuli) play a cardinal role in animal learning, and can be considered as a fundamental tool for developing more autonomous and more adaptive artificial agents. In this work, we provide an exhaustive analysis of a scarcely investigated problem: which kind of IM reinforcement signal is the most suitable for driving the acquisition of multiple skills in the shortest time? To this purpose we implemented an artificial agent with a hierarchical architecture that allows to learn and cache different skills. We tested the system in a setup with continuous states and actions, in particular, with a kinematic robotic arm that has to learn different reaching tasks. We compare the results of different versions of the system driven by several different intrinsic motivation signals. The results show (a) that intrinsic reinforcements purely based on the knowledge of the system are not appropriate to guide the acquisition of multiple skills, and (b) that the stronger the link between the IM signal and the competence of the system, the better the performance

Intrinsic motivation signals for driving the acquisition of multiple tasks: A simulated robotic study

Intrinsic Motivations (i.e motivations not connected to rewardrelated stimuli) drive humans and other biological agents to autonomously learn different skills in absence of any biological pressure or any assigned task. In this paper we investigate which is the best learning signal for driving the training of different tasks in a modular architecture controlling a simulated kinematic robotic arm that has to reach for different objects. We compare the performance of the system varying the Intrinsic Motivation signal and we show how a Task Predictor whose learning process is strictly connected to the competence of the system in the tasks is able to generate the most suitable signal for the autonomous learning of multiple skills

The Algorithmic Numbers in Non-Archimedean Numerical Computing Environments

There are many natural phenomena that can best be described by the use of infinitesimal and infinite numbers (see e.g. [1, 5, 13, 23]. However, until now, the Non-standard techniques have been applied to theoretical models. In this paper we investigate the possibility to implement such models in numerical simulations. First we define the field of Euclidean numbers which is a particular eld of hyperreal numbers. Then, we introduce a set of families of Euclidean numbers, that we have called altogether algorithmic numbers, some of which are inspired by the IEEE 754 standard for floating point numbers. In particular, we suggest three formats which are relevant from the hardware implementation point of view: the Polynomial Algorithmic Numbers, the Bounded Algorithmic Numbers and the Truncated Algorithmic Numbers. In the second part of the paper, we show a few applications of such numbers