Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Marriage, religion and human flourishing: how sustainable is the classic Durkheim thesis in contemporary Europe?

This paper draws on the three waves of the European Values Survey across five countries (Great Britain, Italy, the Netherlands, Northern Ireland, Spain and Sweden) to investigate the relationship between indicators of positive psychology (conceptualised as feelings of happiness and satisfaction with life), religiosity (conceptualised as self-assigned religious affiliation and self-reported religious attendance) and marital status. The results demonstrate that religiosity is, in general, positively correlated with both indicators of positive psychology. Further, across all waves and all countries, the pattern emerges that those respondents who are married are likely to report higher levels of happiness and greater satisfaction in life. These data provide contemporary support for the classic Durkheim thesis linking the two institutions of marriage and religion with human flourishing

Foot health education for people with rheumatoid arthritis : the practitioner's perspective

Background: Patient education is considered to be a key role for podiatrists in the management of patients with rheumatoid arthritis (RA). Patient education has undoubtedly led to improved clinical outcomes, however no attempts have been made to optimise its content or delivery to maximise benefits within the context of the foot affected by rheumatoid arthritis. The aim of this study was to identify the nature and content of podiatrists' foot health education for people with RA. Any potential barriers to its provision were also explored. Methods: A focus group was conducted. The audio dialogue was recorded digitally, transcribed verbatim and analysed using a structured, thematic approach. The full transcription was verified by the focus group as an accurate account of what was said. The thematic analysis framework was verified by members of the research team to ensure validity of the data. Results: Twelve members (all female) of the north west Podiatry Clinical Effectiveness Group for Rheumatology participated. Six overarching themes emerged: (i) the essence of patient education; (ii) the content; (iii) patient-centred approach to content and timing; (iv) barriers to provision; (v) the therapeutic relationship; and (vi) tools of the trade. Conclusion: The study identified aspects of patient education that this group of podiatrists consider most important in relation to its: content, timing, delivery and barriers to its provision. General disease and foot health information in relation to RA together with a potential prognosis for foot health, the role of the podiatrist in management of foot health, and appropriate self-management strategies were considered to be key aspects of content, delivered according to the needs of the individual. Barriers to foot health education provision, including financial constraints and difficulties in establishing effective therapeutic relationships, were viewed as factors that strongly influenced foot health education provision. These data will contribute to the development of a patient-centred, negotiated approach to the provision of foot health education for people with RA

Surface Chemistry of Perfluoropolyethers and Hydrogenated Analogs: Are Studies of Model Compounds Useful?

We have studied adsorption, desorption, and decomposition of ethers on Ru(001), an atomically-smooth metal surface. We have compared diethers with monoethers, and fluorinated ethers with hydrogenated ethers. The number of ether linkages does not strongly influence adsorption bond strength, nor the extent of decomposition. Fluorination does weaken the adsorption bond strength and prevents decomposition. These studies suggest that the surface properties of monomeric ethers can be used to predict properties of oligomeric, and perhaps even polymeric, ethers

Examining links between anxiety, reinvestment and walking when talking by older adults during adaptive gait

Falls by older adults often result in reduced quality of life and debilitating fear of further falls. Stopping walking when talking (SWWT) is a significant predictor of future falls by older adults and is thought to reflect age-related increases in attentional demands of walking. We examine whether SWWT is associated with use of explicit movement cues during locomotion, and evaluate if conscious control (i.e., movement specific reinvestment) is causally linked to falls-related anxiety during a complex walking task. We observed whether twenty-four older adults stopped walking when talking when asked a question during an adaptive gait task. After certain trials, participants completed a visual-spatial recall task regarding walkway features, or answered questions about their movements during the walk. In a subsequent experimental condition, participants completed the walking task under conditions of raised postural threat. Compared to a control group, participants who SWWT reported higher scores for aspects of reinvestment relating to conscious motor processing but not movement self-consciousness. The higher scores for conscious motor processing were preserved when scores representing cognitive function were included as a covariate. There were no group differences in measures of general cognitive function, visual spatial working memory or balance confidence. However, the SWWT group reported higher scores on a test of external awareness when walking, indicating allocation of attention away from task-relevant environmental features. Under conditions of increased threat, participants self-reported significantly greater state anxiety and reinvestment and displayed more accurate responses about their movements during the task. SWWT is not associated solely with age-related cognitive decline or generic increases in age-related attentional demands of walking. SWWT may be caused by competition for phonological resources of working memory associated with consciously processing motor actions and appears to be causally linked with fall-related anxiety and increased vigilance.This research was supported by The Royal Society (IE131576) and British Academy (SG132820)

Insecticide resistance and the future of malaria control in Zambia.

BACKGROUND: In line with the Global trend to improve malaria control efforts a major campaign of insecticide treated net distribution was initiated in 1999 and indoor residual spraying with DDT or pyrethroids was reintroduced in 2000 in Zambia. In 2006, these efforts were strengthened by the President's Malaria Initiative. This manuscript reports on the monitoring and evaluation of these activities and the potential impact of emerging insecticide resistance on disease transmission. METHODS: Mosquitoes were captured daily through a series of 108 window exit traps located at 18 sentinel sites. Specimens were identified to species and analyzed for sporozoites. Adult Anopheles mosquitoes were collected resting indoors and larva collected in breeding sites were reared to F1 and F0 generations in the lab and tested for insecticide resistance following the standard WHO susceptibility assay protocol. Annual cross sectional household parasite surveys were carried out to monitor the impact of the control programme on prevalence of Plasmodium falciparum in children aged 1 to 14 years. RESULTS: A total of 619 Anopheles gambiae s.l. and 228 Anopheles funestus s.l. were captured from window exit traps throughout the period, of which 203 were An. gambiae malaria vectors and 14 An. funestus s.s.. In 2010 resistance to DDT and the pyrethroids deltamethrin, lambda-cyhalothrin and permethrin was detected in both An. gambiae s.s. and An. funestus s.s.. No sporozoites were detected in either species. Prevalence of P. falciparum in the sentinel sites remained below 10% throughout the study period. CONCLUSION: Both An. gambiae s.s. and An. funestus s.s. were controlled effectively with the ITN and IRS programme in Zambia, maintaining a reduced disease transmission and burden. However, the discovery of DDT and pyrethroid resistance in the country threatens the sustainability of the vector control programme

A randomised controlled trial and cost-effectiveness evaluation of "booster" interventions to sustain increases in physical activity in middle-aged adults in deprived urban neighbourhoods

Background: Systematic reviews have identified a range of brief interventions which increase physical activity in previously sedentary people. There is an absence of evidence about whether follow up beyond three months can maintain long term physical activity. This study assesses whether it is worth providing motivational interviews, three months after giving initial advice, to those who have become more active. Methods/Design: Study candidates (n = 1500) will initially be given an interactive DVD and receive two telephone follow ups at monthly intervals checking on receipt and use of the DVD. Only those that have increased their physical activity after three months (n = 600) will be randomised into the study. These participants will receive either a "mini booster" (n = 200), "full booster" (n = 200) or no booster (n = 200). The "mini booster" consists of two telephone calls one month apart to discuss physical activity and maintenance strategies. The "full booster" consists of a face-to-face meeting with the facilitator at the same intervals. The purpose of these booster sessions is to help the individual maintain their increase in physical activity. Differences in physical activity, quality of life and costs associated with the booster interventions, will be measured three and nine months from randomisation. The research will be conducted in 20 of the most deprived neighbourhoods in Sheffield, which have large, ethnically diverse populations, high levels of economic deprivation, low levels of physical activity, poorer health and shorter life expectancy. Participants will be recruited through general practices and community groups, as well as by postal invitation, to ensure the participation of minority ethnic groups and those with lower levels of literacy. Sheffield City Council and Primary Care Trust fund a range of facilities and activities to promote physical activity and variations in access to these between neighbourhoods will make it possible to examine whether the effectiveness of the intervention is modified by access to community facilities. A one-year integrated feasibility study will confirm that recruitment targets are achievable based on a 10% sample.Discussion: The choice of study population, study interventions, brief intervention preceding the study, and outcome measure are discussed

Planet Populations as a Function of Stellar Properties

Exoplanets around different types of stars provide a window into the diverse environments in which planets form. This chapter describes the observed relations between exoplanet populations and stellar properties and how they connect to planet formation in protoplanetary disks. Giant planets occur more frequently around more metal-rich and more massive stars. These findings support the core accretion theory of planet formation, in which the cores of giant planets form more rapidly in more metal-rich and more massive protoplanetary disks. Smaller planets, those with sizes roughly between Earth and Neptune, exhibit different scaling relations with stellar properties. These planets are found around stars with a wide range of metallicities and occur more frequently around lower mass stars. This indicates that planet formation takes place in a wide range of environments, yet it is not clear why planets form more efficiently around low mass stars. Going forward, exoplanet surveys targeting M dwarfs will characterize the exoplanet population around the lowest mass stars. In combination with ongoing stellar characterization, this will help us understand the formation of planets in a large range of environments.Comment: Accepted for Publication in the Handbook of Exoplanet

Explaining participation in the informal economy: An institutional incongruence perspective

Drawing inspiration from institutional theory, a small sub-stream of literature has proposed that participation in the informal economy arises from the lack of alignment of a society’s formal institutions (i.e. its codified laws and regulations) with its informal institutions (i.e. the norms, values and beliefs of its population). To further advance this explanation, this article reports a 2013 Eurobarometer survey involving 27,563 face-to-face interviews across 28 European countries. The finding is that there is a strong association between the degree to which formal and informal institutions are unaligned and participation in the informal economy. The greater is the asymmetry between the formal and informal institutions, the more likely is participation in the informal economy at the individual-, population group- and country-level. A new policy approach for tackling the informal economy which focuses upon reducing this institutional incongruence is then discussed

Recalibrating disparities in perceived and actual balance abilities in older adults: A mixed-methods evaluation of a novel exergaming intervention

This is the final version. Available from the publisher via the DOI in this record.The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.Background: Published reports suggest a disparity between perceived and actual balance abilities, a trait associated with increased fall-risk in older adults. We investigate whether it is possible to 'recalibrate' these disparities using a novel gaming intervention. Methods: We recruited 26 older adults for a 4-week intervention in which they participated in 8-sessions using a novel gaming intervention designed to provide explicit, augmented feedback related to postural control. Measures of perceived balance abilities (Falls Efficacy Scale-International) and actual postural control (limits of stability) were assessed pre- and post-intervention. We used focus groups to elicit the opinions of participants about how the game may have influenced balance abilities and confidence. Results: A stronger alignment was observed between postural control and perceived balance capabilities post-intervention (i.e., significant correlations between Falls Efficacy Scale-International scores and limits of stability which were not present pre-intervention). Also, significant improvements in measures of postural control were observed, with these improvements confined to the aspects of postural control for which the exergame provided explicit, augmented feedback. Qualitative data revealed that the intervention made participants more "aware" of their balance abilities. Conclusions: Our results demonstrate that it is possible to recalibrate the perceptions of older adults relating to their balance abilities through a targeted, short-term intervention. We propose that the post-intervention improvements in postural control may have been, in part, the result of this recalibration; with altered perceptions leading to changes in balance performance. Findings support the application of novel interventions aimed at addressing the psychological factors associated with elderly falls.The Richard Benjamin Trus