Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

Prospective association between late-life physical activity and hospital care utilisation: a 7-year nationwide follow-up study

BACKGROUND: It is still equivocal whether there is a potential role of late-life physical activity in ameliorating the challenges of increasing healthcare expenditure due to the consequence of global population ageing. OBJECTIVE: this study aimed to examine the prospective association between physical activity and subsequent hospital care utilisation in older adults and to explore the optimal dose of physical activity required to reduce hospital care utilisation. DESIGN: this was a prospective cohort study based on the data from the Taiwan 2005 National Health Interview Survey, which were linked to the 2005-12 claims data from the National Health Insurance system. PARTICIPANTS: 1,760 older adults aged 65 or more. METHODS: the frequency, duration and intensity for physical activity were assessed, and total physical activity energy expenditure was estimated. The average annualised hospital care utilisation for the period 2006 through 2012, including number of hospitalisations, number of days in hospital and the costs of hospitalisation, were calculated. RESULTS: older adults engaging in at least moderate volume of physical activity (≥1,000 kcal/week) experienced fewer subsequent hospital admissions and fewer days in hospital than did sedentary individuals, after adjusting for covariates. Trends for reduced hospitalisation costs were also found. These associations persisted in sensitivity analyses, including tests of reverse causation. CONCLUSION: this study has provided evidence that older adults who are at least moderately active may minimise utilisation of hospital care services. The findings highlight the importance of maintaining a physically active lifestyle in later life

Samlande och visualiserande av verksamhetsstyrande data för en digital marknadsföringsbyrå

I dagens läge, med stor konkurrens mellan företag i samma bransch är det ytterst viktigt för företag att veta hur de når sina målgrupper. I detta examensarbete diskuteras vikten av verksamhetsstyrande data, hur det samlats och visualiserats tidigare jämfört med idag, och hur företag använder sig av den för att göra beslut och ändringar i sina strategier. Arbetet hänvisar till flera artiklar och intervjuer med experter inom visualisering av data, samt datadriven beslutsfattning inom företag. Målet med arbetet är att ha en fungerande nätverksapplikation, skriven i Node.js, som automatiskt sköter samlande av data från olika molntjänster ett företag använder. Utöver samlandet skall applikationen kunna utföra räkningar på data och skicka den vidare till en tjänst som visualiserar den i olika widgets på en dashboard.With increasing competition between companies within the same area, it is becoming more important than ever for companies to know how to reach their target groups. This thesis focus on the importance of performance related data, how it has been gathered and visualized earlier compared to today, and how companies use it to make data-based decisions for their strategies. Articles and interviews with experts in the fields of data visualization and data driven management are reviewed in the thesis. The goal is to have a working network application that gathers data from several online services in use by the company. In addition to gathering, the application will also perform calculations and format the data in such a way that it is easy to visualize using widgets on dashboards

Physical activity, smoking, and the incidence of clinically diagnosed insomnia

OBJECTIVE: This study was designed to examine the independent and combined associations of physical activity and smoking on the incidence of doctor-diagnosed insomnia using a nationally representative sample over seven years, taking into account other relevant covariates. METHODS: Participants aged 18 years or older in the 2005 Taiwan National Health Interview Survey (NHIS) with links to National Health Insurance (NHI) claim data between 2005 and 2012 and without diagnosed insomnia before 2005, were selected into this study (n = 12,728). Participants were classified as having insomnia with International Classification of Diseases, Ninth Revision (ICD-9) CM codes 307.41, 307.42, or 780.52. Self-reported smoking status and frequency, duration, and types of leisure-time and non−leisure-time physical activities were collected. Metabolic equivalent (MET) intensity levels for each activity were assigned, and weekly energy expenditure of each activity was calculated and summed. RESULTS: Inactive participants had a higher risk of incident insomnia [hazard ratio (HR) = 1.22, 95% confidence interval (CI) = 1.06–1.42, p = 0.007] than the active group, and ever-smokers were more likely to have incident insomnia than never smokers (HR = 1.45, 95% CI = 1.20–1.76, p < 0.001). Compared with the nonsmoker/active group, the ever-smoker/inactive group had a higher risk of incident insomnia (HR = 1.78, 95% CI = 1.41–2.25, p < 0.001). Sensitivity analyses excluding individuals diagnosed with other sleep disorders or mental disorders yielded similar results, with the ever-smoker/inactive group having the highest risk of insomnia. CONCLUSIONS: Inactive adults and smokers are at higher risk for incident insomnia, highlighting the importance of a healthy lifestyle and pointing to strategies such as encouraging smoking cessation and physical activity to avoid insomnia among adults

Identification and validation of oncologic miRNA biomarkers for Luminal A-like breast cancer

Introduction: Breast cancer is a common disease with distinct tumor subtypes phenotypically characterized by ER and HER2/neu receptor status. MiRNAs play regulatory roles in tumor initiation and progression, and altered miRNA expression has been demonstrated in a variety of cancer states presenting the potential for exploitation as cancer biomarkers. Blood provides an excellent medium for biomarker discovery. This study investigated systemic miRNAs differentially expressed in Luminal A-like (ER+PR+HER2/neu-) breast cancer and their effectiveness as oncologic biomarkers in the clinical setting. Methods: Blood samples were prospectively collected from patients with Luminal A-like breast cancer (n=54) and controls (n=56). RNA was extracted, reverse transcribed and subjected to microarray analysis (n=10 Luminal A-like; n=10 Control). Differentially expressed miRNAs were identified by artificial neural network (ANN) data-mining algorithms. Expression of specific miRNAs was validated by RQ-PCR (n=44 Luminal A; n=46 Control) and potential relationships between circulating miRNA levels and clinicopathological features of breast cancer were investigated. Results: Microarray analysis identified 76 differentially expressed miRNAs. ANN revealed 10 miRNAs for further analysis ( miR-19b, miR-29a, miR-93, miR-181a, miR-182, miR-223, miR-301a, miR-423-5p, miR-486-5 and miR-652 ). The biomarker potential of 4 miRNAs ( miR-29a, miR-181a , miR-223 and miR-652 ) was confirmed by RQ-PCR, with significantly reduced expression in blood of women with Luminal A-like breast tumors compared to healthy controls (p=0.001, 0.004, 0.009 and 0.004 respectively). Binary logistic regression confirmed that combination of 3 of these miRNAs ( miR-29a, miR-181a and miR-652 ) could reliably differentiate between cancers and controls with an AUC of 0.80. Conclusion: This study provides insight into the underlying molecular portrait of Luminal A-like breast cancer subtype. From an initial 76 miRNAs, 4 were validated with altered expression in the blood of women with Luminal A-like breast cancer. The expression profiles of these 3 miRNAs, in combination with mammography, has potential to facilitate accurate subtype- specific breast tumor detection

Compartment-Specific and Sequential Role of MyD88 and CARD9 in Chemokine Induction and Innate Defense during Respiratory Fungal Infection

Aspergillus fumigatus forms ubiquitous airborne conidia that humans inhale on a daily basis. Although respiratory fungal infection activates the adaptor proteins CARD9 and MyD88 via C-type lectin, Toll-like, and interleukin-1 family receptor signals, defining the temporal and spatial pattern of MyD88- and CARD9-coupled signals in immune activation and fungal clearance has been difficult to achieve. Herein, we demonstrate that MyD88 and CARD9 act in two discrete phases and in two cellular compartments to direct chemokine- and neutrophil-dependent host defense. The first phase depends on MyD88 signaling because genetic deletion of MyD88 leads to delayed induction of the neutrophil chemokines CXCL1 and CXCL5, delayed neutrophil lung trafficking, and fatal pulmonary damage at the onset of respiratory fungal infection. MyD88 expression in lung epithelial cells restores rapid chemokine induction and neutrophil recruitment via interleukin-1 receptor signaling. Exogenous CXCL1 administration reverses murine mortality in MyD88-deficient mice. The second phase depends predominately on CARD9 signaling because genetic deletion of CARD9 in radiosensitive hematopoietic cells interrupts CXCL1 and CXCL2 production and lung neutrophil recruitment beyond the initial MyD88-dependent phase. Using a CXCL2 reporter mouse, we show that lung-infiltrating neutrophils represent the major cellular source of CXCL2 during CARD9-dependent recruitment. Although neutrophil-intrinsic MyD88 and CARD9 function are dispensable for neutrophil conidial uptake and killing in the lung, global deletion of both adaptor proteins triggers rapidly progressive invasive disease when mice are challenged with an inoculum that is sub-lethal for single adapter protein knockout mice. Our findings demonstrate that distinct signal transduction pathways in the respiratory epithelium and hematopoietic compartment partially overlap to ensure optimal chemokine induction, neutrophil recruitment, and fungal clearance within the respiratory tract

Pleiotropic functions of the tumor- and metastasis-suppressing Matrix Metalloproteinase-8 in mammary cancer in MMTV-PyMT transgenic mice

Matrix metalloproteinase-8 (MMP-8; neutrophil collagenase) is an important regulator of innate immunity which has onco-suppressive actions in numerous tumor types

Facile Synthesis of High Quality Graphene Nanoribbons

Graphene nanoribbons have attracted attention for their novel electronic and spin transport properties1-6, and because nanoribbons less than 10 nm wide have a band gap that can be used to make field effect transistors. However, producing nanoribbons of very high quality, or in high volumes, remains a challenge. Here, we show that pristine few-layer nanoribbons can be produced by unzipping mildly gas-phase oxidized multiwalled carbon nanotube using mechanical sonication in an organic solvent. The nanoribbons exhibit very high quality, with smooth edges (as seen by high-resolution transmission electron microscopy), low ratios of disorder to graphitic Raman bands, and the highest electrical conductance and mobility reported to date (up to 5e2/h and 1500 cm2/Vs for ribbons 10-20 nm in width). Further, at low temperature, the nanoribbons exhibit phase coherent transport and Fabry-Perot interference, suggesting minimal defects and edge roughness. The yield of nanoribbons was ~2% of the starting raw nanotube soot material, which was significantly higher than previous methods capable of producing high quality narrow nanoribbons1. The relatively high yield synthesis of pristine graphene nanoribbons will make these materials easily accessible for a wide range of fundamental and practical applications.Comment: Nature Nanotechnology in pres

Patient-centric trials for therapeutic development in precision oncology

An enhanced understanding of the molecular pathology of disease gained from genomic studies is facilitating the development of treatments that target discrete molecular subclasses of tumours. Considerable associated challenges include how to advance and implement targeted drug-development strategies. Precision medicine centres on delivering the most appropriate therapy to a patient on the basis of clinical and molecular features of their disease. The development of therapeutic agents that target molecular mechanisms is driving innovation in clinical-trial strategies. Although progress has been made, modifications to existing core paradigms in oncology drug development will be required to realize fully the promise of precision medicine

Most vital segment barriers

We study continuous analogues of "vitality" for discrete network flows/paths, and consider problems related to placing segment barriers that have highest impact on a flow/path in a polygonal domain. This extends the graph-theoretic notion of "most vital arcs" for flows/paths to geometric environments. We give hardness results and efficient algorithms for various versions of the problem, (almost) completely separating hard and polynomially-solvable cases