Multi-Task Policy Search for Robotics

© 2014 IEEE.Learning policies that generalize across multiple tasks is an important and challenging research topic in reinforcement learning and robotics. Training individual policies for every single potential task is often impractical, especially for continuous task variations, requiring more principled approaches to share and transfer knowledge among similar tasks. We present a novel approach for learning a nonlinear feedback policy that generalizes across multiple tasks. The key idea is to define a parametrized policy as a function of both the state and the task, which allows learning a single policy that generalizes across multiple known and unknown tasks. Applications of our novel approach to reinforcement and imitation learning in realrobot experiments are shown

Multi-Task Policy Search

Learning policies that generalize across multiple tasks is an important and challenging research topic in reinforcement learning and robotics. Training individual policies for every single potential task is often impractical, especially for continuous task variations, requiring more principled approaches to share and transfer knowledge among similar tasks. We present a novel approach for learning a nonlinear feedback policy that generalizes across multiple tasks. The key idea is to define a parametrized policy as a function of both the state and the task, which allows learning a single policy that generalizes across multiple known and unknown tasks. Applications of our novel approach to reinforcement and imitation learning in real-robot experiments are shown

Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning

Over the last years, there has been substantial progress in robust manipulation in unstructured environments. The long-term goal of our work is to get away from precise, but very expensive robotic systems and to develop affordable, potentially imprecise, self-adaptive manipulator systems that can interactively perform tasks such as playing with children. In this paper, we demonstrate how a low-cost off-the-shelf robotic system can learn closed-loop policies for a stacking task in only a handful of trials-from scratch. Our manipulator is inaccurate and provides no pose feedback. For learning a controller in the work space of a Kinect-style depth camera, we use a model-based reinforcement learning technique. Our learning method is data efficient, reduces model bias, and deals with several noise sources in a principled way during long-term planning. We present a way of incorporating state-space constraints into the learning process and analyze the learning gain by exploiting the sequential structure of the stacking task

Treatment eligibility and retention in clinical HIV care: A regression discontinuity study in South Africa

BACKGROUND: Loss to follow-up is high among HIV patients not yet receiving antiretroviral therapy (ART). Clinical trials have demonstrated the clinical efficacy of early ART; however, these trials may miss an important real-world consequence of providing ART at diagnosis: its impact on retention in care. METHODS AND FINDINGS: We examined the effect of immediate (versus deferred) ART on retention in care using a regression discontinuity design. The analysis included all patients (N = 11,306) entering clinical HIV care with a first CD4 count between 12 August 2011 and 31 December 2012 in a public-sector HIV care and treatment program in rural South Africa. Patients were assigned to immediate versus deferred ART eligibility, as determined by a CD4 count < 350 cells/μl, per South African national guidelines. Patients referred to pre-ART care were instructed to return every 6 months for CD4 monitoring. Patients initiated on ART were instructed to return at 6 and 12 months post-initiation and annually thereafter for CD4 and viral load monitoring. We assessed retention in HIV care at 12 months, as measured by the presence of a clinic visit, lab test, or ART initiation 6 to 18 months after initial CD4 test. Differences in retention between patients presenting with CD4 counts just above versus just below the 350-cells/μl threshold were estimated using local linear regression models with a data-driven bandwidth and with the algorithm for selecting the bandwidth chosen ex ante. Among patients with CD4 counts close to the 350-cells/μl threshold, having an ART-eligible CD4 count (<350 cells/μl) was associated with higher 12-month retention than not having an ART-eligible CD4 count (50% versus 32%), an intention-to-treat risk difference of 18 percentage points (95% CI 11 to 23; p < 0.001). The decision to start ART was determined by CD4 count for one in four patients (25%) presenting close to the eligibility threshold (95% CI 20% to 31%; p < 0.001). In this subpopulation, having an ART-eligible CD4 count was associated with higher 12-month retention than not having an ART-eligible CD4 count (91% versus 21%), a complier causal risk difference of 70 percentage points (95% CI 42 to 98; p < 0.001). The major limitations of the study are the potential for limited generalizability, the potential for outcome misclassification, and the absence of data on longer-term health outcomes. CONCLUSIONS: Patients who were eligible for immediate ART had dramatically higher retention in HIV care than patients who just missed the CD4-count eligibility cutoff. The clinical and population health benefits of offering immediate ART regardless of CD4 count may be larger than suggested by clinical trials

Building governance and energy efficiency: Mapping the interdisciplinary challenge

Improving the energy efficiency of multi-owned properties (MoPs)—commonly known as apartment or condominium buildings—is central to the achievement of European energy targets. However, little work to date has focused on how to facilitate retrofit in this context. Drawing on interdisciplinary Social Sciences and Humanities expertise in academia, policy and practice, this chapter posits that decision-making processes within MoPs might provide a key to the retrofit challenge. Existing theories or models of decision-making, applied in the MoP context, might help to explain how collective retrofit decisions are taken—or overlooked. Insights from case studies and practitioners are also key. Theories of change might then be employed to develop strategies to facilitate positive retrofit decisions. The chapter maps the issues and sets an agenda for further interdisciplinary research in this novel area

Migrations and habitat use of the smooth hammerhead shark (Sphyrna zygaena) in the Atlantic Ocean

The smooth hammerhead shark, Sphyrna zygaena, is a cosmopolitan semipelagic shark captured as bycatch in pelagic oceanic fisheries, especially pelagic longlines targeting swordfish and/or tunas. From 2012 to 2016, eight smooth hammerheads were tagged with Pop-up Satellite Archival Tags in the inter-tropical region of the Northeast Atlantic Ocean, with successful transmissions received from seven tags (total of 319 tracking days). Results confirmed the smooth hammerhead is a highly mobile species, as the longest migration ever documented for this species (> 6600 km) was recorded. An absence of a diel vertical movement behavior was noted, with the sharks spending most of their time at surface waters (0-50 m) above 23 degrees C. The operating depth of the pelagic long-line gear was measured with Minilog Temperature and Depth Recorders, and the overlap with the species vertical distribution was calculated. The overlap is taking place mainly during the night and is higher for juveniles (similar to 40% of overlap time). The novel information presented can now be used to contribute to the provision of sustainable management tools and serve as input for Ecological Risk Assessments for smooth hammerheads caught in Atlantic pelagic longline fisheries.Oceanario de Lisboa through Project "SHARK-TAG: Migrations and habitat use of the smooth hammerhead shark in the Atlantic Ocean"; Investigador-FCT from the Portuguese Foundation for Science and Technology (FCT, Fundacao para a Ciencia e Tecnologia) [Ref: IF/00253/2014]; EU European Social Fund; Programa Operacional Potencial Human

Diagnosing Dementia in the Clinical Setting: Can Amyloid PET Provide Additional Value Over Cerebrospinal Fluid?

Cerebrospinal fluid (CSF) measures of amyloid and tau are the first-line Alzheimer's disease biomarkers in many clinical centers. We assessed if and when the addition of amyloid PET following CSF measurements provides added diagnostic value. Twenty patients from a cognitive clinic, who had undergone detailed assessment including CSF measures, went on to have amyloid PET. The treating neurologist's working diagnosis, and degree of diagnostic certainty, was assessed both before and after the PET. Amyloid PET changed the diagnosis in 7/20 cases. Amyloid PET can provide added diagnostic value, particularly in young-onset, atypical dementias, where CSF results are borderline and diagnostic uncertainty remains

A meta-analytic review of stand-alone interventions to improve body image

Objective Numerous stand-alone interventions to improve body image have been developed. The present review used meta-analysis to estimate the effectiveness of such interventions, and to identify the specific change techniques that lead to improvement in body image. Methods The inclusion criteria were that (a) the intervention was stand-alone (i.e., solely focused on improving body image), (b) a control group was used, (c) participants were randomly assigned to conditions, and (d) at least one pretest and one posttest measure of body image was taken. Effect sizes were meta-analysed and moderator analyses were conducted. A taxonomy of 48 change techniques used in interventions targeted at body image was developed; all interventions were coded using this taxonomy. Results The literature search identified 62 tests of interventions (N = 3,846). Interventions produced a small-to-medium improvement in body image (d+ = 0.38), a small-to-medium reduction in beauty ideal internalisation (d+ = -0.37), and a large reduction in social comparison tendencies (d+ = -0.72). However, the effect size for body image was inflated by bias both within and across studies, and was reliable but of small magnitude once corrections for bias were applied. Effect sizes for the other outcomes were no longer reliable once corrections for bias were applied. Several features of the sample, intervention, and methodology moderated intervention effects. Twelve change techniques were associated with improvements in body image, and three techniques were contra-indicated. Conclusions The findings show that interventions engender only small improvements in body image, and underline the need for large-scale, high-quality trials in this area. The review identifies effective techniques that could be deployed in future interventions

Millisecond Oscillations in X-Ray Binaries

The first millisecond X-ray variability phenomena from accreting compact objects have recently been discovered with the Rossi X-ray Timing Explorer. Three new phenomena are observed from low-mass X-ray binaries containing low-magnetic-field neutron stars: millisecond pulsations, burst oscillations and kiloHertz quasi-periodic oscillations. Models for these new phenomena involve the neutron star spin, and orbital motion closely around the neutron star and rely explicitly on our understanding of strong gravity and dense matter. I review the observations of these new neutron-star phenomena and possibly related ones in black-hole candidates, and describe the attempts to use them to perform measurements of fundamental physical interest in these systems.Comment: 40 pages, 17 figures, 4 tables - submitted to the Annual Review of Astronomy and Astrophysics; to appear September 200