Reinforcement Learning: A Survey

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.Comment: See http://www.jair.org/ for any accompanying file

Phosphorus Immobilization in Poultry Litter and Litter-amended soils with Aluminum, Calcium and Iron amendments

Arkansas produces approximately one billion broilers each year. Phosphorous (P) runoff from fields receiving poultry litter is believed to be one of the primary factors affecting water quality in Northwest Arkansas. Poultry litter contains approximately 20 g P kg-1, of which about 2 g P kg-1 is water soluble. Soils that have received repeated heavy applications of litter may have water soluble P contents of as high as 10 mg P Kg-1 soil. The objective of this study was to determine if soluble P levels could be reduced in poultry litter and litter-amended soils with Al,Ca, and/or Fe amendments. Poultry litter was amended with alum, sodium aluminate, quick lime, slaked lime, calcitic limestone, dolomitic limestone, gypsum, ferrous chloride, ferric chloride, ferrous sulfate and ferric sulfate, and incubated in the dark at 25°C for one week. Three soils which had been excessively fertilized with poultry litter were amended with alum, ferrous sulfate, calcitic limestone, gypsum and slaked lime and incubated for 4 weeks at 25 °C. In the litter studies, the Ca treatments were tested with and without CaF2 additions in an attempt to precipitate fluorapatite. At the end of the incubation period, the litter and soils were extracted with deionized water and soluble reactive P (SRP) was determined. SRP levels in the poultry litter were reduced from over 2,000 mg P kg-1 litter to less than 1 mg P kg-1 litter with the addition of alum, quick lime, slaked lime, ferrous chloride, ferric chloride, ferrous sulfate and ferric sulfate under favorable pH conditions. S.RP levels in the soils were reduced from approximately 5 mg P Kg-1 soil to less than 0.05 mg P Kg-1 soil with the addition of alum and ferrous sulfate under favorable pH conditions. Gypsum and sodium aluminate reduced SRP levels in litter by 50 to 60 percent while calcitic and dolomitic limestone were even less effective. In soils, the Ca amendments were less effective than the Al and Fe amendments, although slaked lime was effective at high pH. The results of these studies suggest that treating litter and excessively fertilized soils with some of these compounds, particularly alum, could significantly reduce the amount of SRP in runoff from littered pastures. Therefore, chemical additions to reduce SRP in litter and soil may be a best management practice in situations where eutrophication of adjacent water bodies due to P runoff has been identified. Preliminary calculations indicate that this .p ractice may be economically feasible. However, more research is needed to determine any beneficial and/or detrimental aspects of this practice

Palomar 13: a velocity dispersion inflated by binaries ?

Recently, combining radial velocities from Keck/HIRES echelle spectra with published proper motion membership probabilities, Cote et al (2002) observed a sample of 21 stars, probable members of Palomar 13, a globular cluster in the Galactic halo. Their projected velocity dispersion sigma_p = 2.2 +/-0.4 km/s gives a mass-to-light ratio M/L_V = 40 +24/-17, about one order of magnitude larger than the usual estimate for globular clusters. We present here radial velocities measured from three different CCD frames of commissioning observations obtained with the new ESO/VLT instrument FLAMES (Fibre Large Array Multi Element Spectrograph). From these data, now publicly available, we measure the homogeneous radial velocities of eight probable members of this globular cluster. A new projected velocity dispersion sigma_p = 0.6-0.9 +/-0.3 km/s implies Palomar 13 mass-to-light ratio M/L_V = 3-7, similar to the usual value for globular clusters. We discuss briefly the two most obvious reasons for the previous unusual mass-to-light ratio finding: binaries, now clearly detected, and more homogeneous data from the multi-fibre FLAMES spectrograph.Comment: 9 pages, 2 Postscript figure

Entanglement entropy of random quantum critical points in one dimension

For quantum critical spin chains without disorder, it is known that the entanglement of a segment of N>>1 spins with the remainder is logarithmic in N with a prefactor fixed by the central charge of the associated conformal field theory. We show that for a class of strongly random quantum spin chains, the same logarithmic scaling holds for mean entanglement at criticality and defines a critical entropy equivalent to central charge in the pure case. This effective central charge is obtained for Heisenberg, XX, and quantum Ising chains using an analytic real-space renormalization group approach believed to be asymptotically exact. For these random chains, the effective universal central charge is characteristic of a universality class and is consistent with a c-theorem.Comment: 4 pages, 3 figure

Towards Informative Path Planning for Acoustic SLAM

Acoustic scene mapping is a challenging task as microphone arrays can often localize sound sources only in terms of their directions. Spatial diversity can be exploited constructively to infer source-sensor range when using microphone arrays installed on moving platforms, such as robots. As the absolute location of a moving robot is often unknown in practice, Acoustic Simultaneous Localization And Mapping (a-SLAM) is required in order to localize the moving robot’s positions and jointly map the sound sources. Using a novel a-SLAM approach, this paper investigates the impact of the choice of robot paths on source mapping accuracy. Simulation results demonstrate that a-SLAM performance can be improved by informatively planning robot paths

Unified model for vortex-string network evolution

We describe and numerically test the velocity-dependent one-scale (VOS) string evolution model, a simple analytic approach describing a string network with the averaged correlation length and velocity. We show that it accurately reproduces the large-scale behaviour (in particular the scaling laws) of numerical simulations of both Goto-Nambu and field theory string networks. We explicitly demonstrate the relation between the high-energy physics approach and the damped and non-relativistic limits which are relevant for condensed matter physics. We also reproduce experimental results in this context and show that the vortex-string density is significantly reduced by loop production, an effect not included in the usual `coarse-grained' approach.Comment: 5 pages; v2: cosmetic changes, version to appear in PR

Numerical studies of a one-dimensional 3-spin spin-glass model with long-range interactions

We study a p-spin spin-glass model to understand if the finite-temperature glass transition found in the mean-field regime of p-spin models, and used to model the behavior of structural glasses, persists in the non-mean-field regime. By using a 3-spin spin-glass model with long-range power-law diluted interactions we are able to continuously tune the (effective) space dimension via the exponent of the interactions. Monte Carlo simulations of the spin-glass susceptibility and the two-point finite-size correlation length show that deep in the non-mean-field regime the finite-temperature transition is lost, whereas this is not the case in the mean-field regime, in agreement with the prediction of Moore and Drossel [Phys. Rev. Lett. 89, 217202 (2002)] that 3-spin models are in the same universality class as an Ising spin glass in a magnetic field. However, slightly in the non-mean-field region, we find an apparent transition in the 3-spin model, in contrast to results for the Ising spin glass in a field. This may indicate that even larger sizes are needed to probe the asymptotic behavior in this region.Comment: 8 pages, 9 figures, 1 tabl

Probing for Binding Regions of the FtsZ Protein Surface through Site-Directed Insertions: Discovery of Fully Functional FtsZ-Fluorescent Proteins

FtsZ, a bacterial tubulin homologue, is a cytoskeletal protein that assembles into protofilaments that are one subunit thick. These protofilaments assemble further to form a “Z ring” at the center of prokaryotic cells. The Z ring generates a constriction force on the inner membrane and also serves as a scaffold to recruit cell wall remodeling proteins for complete cell division in vivo. One model of the Z ring proposes that protofilaments associate via lateral bonds to form ribbons; however, lateral bonds are still only hypothetical. To explore potential lateral bonding sites, we probed the surface of Escherichia coli FtsZ by inserting either small peptides or whole fluorescent proteins (FPs). Among the four lateral surfaces on FtsZ protofilaments, we obtained inserts on the front and back surfaces that were functional for cell division. We concluded that these faces are not sites of essential interactions. Inserts at two sites, G124 and R174, located on the left and right surfaces, completely blocked function, and these sites were identified as possible sites for essential lateral interactions. However, the insert at R174 did not interfere with association of protofilaments into sheets and bundles in vitro. Another goal was to find a location within FtsZ that supported insertion of FP reporter proteins while allowing the FtsZ-FPs to function as the sole source of FtsZ. We discovered one internal site, G55-Q56, where several different FPs could be inserted without impairing function. These FtsZ-FPs may provide advances for imaging Z-ring structure by superresolution techniques. IMPORTANCE One model for the Z-ring structure proposes that protofilaments are assembled into ribbons by lateral bonds between FtsZ subunits. Our study excluded the involvement of the front and back faces of the protofilament in essential interactions in vivo but pointed to two potential lateral bond sites, on the right and left sides. We also identified an FtsZ loop where various fluorescent proteins could be inserted without blocking function; these FtsZ-FPs functioned as the sole source of FtsZ. This advance provides improved tools for all fluorescence imaging of the Z ring and may be especially important for superresolution imaging

Recognising Desire: A psychosocial approach to understanding education policy implementation and effect

It is argued that in order to understand the ways in which teachers experience their work - including the idiosyncratic ways in which they respond to and implement mandated education policy - it is necessary to take account both of sociological and of psychological issues. The paper draws on original research with practising and beginning teachers, and on theories of social and psychic induction, to illustrate the potential benefits of this bipartisan approach for both teachers and researchers. Recognising the significance of (but somewhat arbitrary distinction between) structure and agency in teachers’ practical and ideological positionings, it is suggested that teachers’ responses to local and central policy changes are governed by a mix of pragmatism, social determinism and often hidden desires. It is the often underacknowledged strength of desire that may tip teachers into accepting and implementing policies with which they are not ideologically comfortable