Smoothing Policies and Safe Policy Gradients

Policy gradient algorithms are among the best candidates for the much anticipated application of reinforcement learning to real-world control tasks, such as the ones arising in robotics. However, the trial-and-error nature of these methods introduces safety issues whenever the learning phase itself must be performed on a physical system. In this paper, we address a specific safety formulation, where danger is encoded in the reward signal and the learning agent is constrained to never worsen its performance. By studying actor-only policy gradient from a stochastic optimization perspective, we establish improvement guarantees for a wide class of parametric policies, generalizing existing results on Gaussian policies. This, together with novel upper bounds on the variance of policy gradient estimators, allows to identify those meta-parameter schedules that guarantee monotonic improvement with high probability. The two key meta-parameters are the step size of the parameter updates and the batch size of the gradient estimators. By a joint, adaptive selection of these meta-parameters, we obtain a safe policy gradient algorithm

Transfer from Multiple MDPs

Transfer reinforcement learning (RL) methods leverage on the experience collected on a set of source tasks to speed-up RL algorithms. A simple and effective approach is to transfer samples from source tasks and include them into the training set used to solve a given target task. In this paper, we investigate the theoretical properties of this transfer method and we introduce novel algorithms adapting the transfer process on the basis of the similarity between source and target tasks. Finally, we report illustrative experimental results in a continuous chain problem.Comment: 201

Unimodal Thompson Sampling for Graph-Structured Arms

We study, to the best of our knowledge, the first Bayesian algorithm for unimodal Multi-Armed Bandit (MAB) problems with graph structure. In this setting, each arm corresponds to a node of a graph and each edge provides a relationship, unknown to the learner, between two nodes in terms of expected reward. Furthermore, for any node of the graph there is a path leading to the unique node providing the maximum expected reward, along which the expected reward is monotonically increasing. Previous results on this setting describe the behavior of frequentist MAB algorithms. In our paper, we design a Thompson Sampling-based algorithm whose asymptotic pseudo-regret matches the lower bound for the considered setting. We show that -as it happens in a wide number of scenarios- Bayesian MAB algorithms dramatically outperform frequentist ones. In particular, we provide a thorough experimental evaluation of the performance of our and state-of-the-art algorithms as the properties of the graph vary

Eating at Home and "Dining" Out? Commensalities in the Neolithic and Late Chalcolithic in the Near East

This paper attempts to draw a picture of different kinds of commensalities in the Near Eastern Pottery Neolithic (7th millennium BC) through an analysis of consumption vessels. The case study will be the Syrian and Turkish regions of the Northern Levant. I shall underline the strong symbolic function of vessels in distinguishing commensal events and argue that the basic role of commensality remains largely unmodified until the end of the Ubaid period (2nd half of 5th millennium BC). The beginning of the Late Chalcolithic then marks a major change. At this point, the development of different types of commensalities leads to a decrease in the role of pottery as symbolic marker of commensal events

Efficient single-spatial-mode periodically-poled KTiOPO_4 waveguide source for high-dimensional entanglement-based quantum key distribution

We demonstrate generation of high-purity photon pairs at 1560 nm in a single spatial mode from a periodically-poled KTiOPO_4 (PPKTP) waveguide. With nearly lossless spectral filtering, the PPKTP waveguide source shows approximately 80 % single-mode fiber coupling efficiency and is well suited for high-dimensional time-energy entanglement-based quantum key distribution. Using high-count-rate self-differencing InGaAs single-photon avalanche photodiodes configured with either square or sinusoidal gating, we achieve > 1 Mbit/s raw key generation with 3 bits-per-photon encoding, and, to the best of our knowledge, the highest reported Franson quantum-interference visibility of 98.2 % without subtraction of accidental coincidences

Italian foreign and security policy in a state of reliability crisis?

This article focuses on Italian foreign and security policy (IFSP). It looks at three examples of the country's policy-making which reveal its poor results as a security provider, namely: Italy's tardy reaction to the violence in Libya in 2011, its prompt reaction to the Lebanon crisis in 2006, and its efforts to be included in the diplomatic directorate, the P5+1, approaching relations with Iran in 2009. The article considers whether government action has bolstered the reliability of IFSP and also discusses the country's FSP in terms of its basic differences from that of its partners in the European Union, France, Britain and Germany, envisaging how Italy could react to build more credibility. Italy's policy is observed through a three-pronged analytical framework enriched by concepts of the logic of expected consequences. The article concludes that IFSP is predictable, but it must still reveal that it is reliable, and explains why this is the case