A Survey on Policy Search for Robotics

Policy search is a subfield in reinforcement learning which focuses on finding good parameters for a given policy parametrization. It is well suited for robotics as it can cope with high-dimensional state and action spaces, one of the main challenges in robot learning. We review recent successes of both model-free and model-based policy search in robot learning. Model-free policy search is a general approach to learn policies based on sampled trajectories. We classify model-free methods based on their policy evaluation strategy, policy update strategy, and exploration strategy and present a unified view on existing algorithms. Learning a policy is often easier than learning an accurate forward model, and, hence, model-free methods are more frequently used in practice. However, for each sampled trajectory, it is necessary to interact with the * Both authors contributed equally. robot, which can be time consuming and challenging in practice. Modelbased policy search addresses this problem by first learning a simulator of the robot’s dynamics from data. Subsequently, the simulator generates trajectories that are used for policy learning. For both modelfree and model-based policy search methods, we review their respective properties and their applicability to robotic systems

A Novel Unsupervised Method to Identify Genes Important in the Anti-viral Response: Application to Interferon/Ribavirin in Hepatitis C Patients

Background: Treating hepatitis C with interferon/ribavirin results in a varied response in terms of decrease in viral titer and ultimate outcome. Marked responders have a sharp decline in viral titer within a few days of treatment initiation, whereas in other patients there is no effect on the virus (poor responders). Previous studies have shown that combination therapy modifies expression of hundreds of genes in vitro and in vivo. However, identifying which, if any, of these genes have a role in viral clearance remains challenging. Aims: The goal of this paper is to link viral levels with gene expression and thereby identify genes that may be responsible for early decrease in viral titer. Methods: Microarrays were performed on RNA isolated from PBMC of patients undergoing interferon/ribavirin therapy. Samples were collected at pre-treatment (day 0), and 1, 2, 7, 14 and 28 days after initiating treatment. A novel method was applied to identify genes that are linked to a decrease in viral titer during interferon/ribavirin treatment. The method uses the relationship between inter-patient gene expression based proximities and inter-patient viral titer based proximities to define the association between microarray gene expression measurements of each gene and viral-titer measurements. Results: We detected 36 unique genes whose expressions provide a clustering of patients that resembles viral titer based clustering of patients. These genes include IRF7, MX1, OASL and OAS2, viperin and many ISG's of unknown function. Conclusion: The genes identified by this method appear to play a major role in the reduction of hepatitis C virus during the early phase of treatment. The method has broad utility and can be used to analyze response to any group of factors influencing biological outcome such as antiviral drugs or anti-cancer agents where microarray data are available. © 2007 Brodsky et al

An introduction to Graph Data Management

A graph database is a database where the data structures for the schema and/or instances are modeled as a (labeled)(directed) graph or generalizations of it, and where querying is expressed by graph-oriented operations and type constructors. In this article we present the basic notions of graph databases, give an historical overview of its main development, and study the main current systems that implement them

Acute WNT signalling activation perturbs differentiation within the adult stomach and rapidly leads to tumour formation

A role for WNT signalling in gastric carcinogenesis has been suggested due to two major observations. First, patients with germline mutations in adenomatous polyposis coli (APC) are susceptible to stomach polyps and second, in gastric cancer, WNT activation confers a poor prognosis. However, the functional significance of deregulated WNT signalling in gastric homoeostasis and cancer is still unclear. In this study we have addressed this by investigating the immediate effects of WNT signalling activation within the stomach epithelium. We have specifically activated the WNT signalling pathway within the mouse adult gastric epithelium via deletion of either glycogen synthase kinase 3 (GSK3) or APC or via expression of a constitutively active β-catenin protein. WNT pathway deregulation dramatically affects stomach homoeostasis at very short latencies. In the corpus, there is rapid loss of parietal cells with fundic gland polyp (FGP) formation and adenomatous change, which are similar to those observed in familial adenomatous polyposis. In the antrum, adenomas occur from 4 days post-WNT activation. Taken together, these data show a pivotal role for WNT signalling in gastric homoeostasis, FGP formation and adenomagenesis. Loss of the parietal cell population and corresponding FGP formation, an early event in gastric carcinogenesis, as well as antral adenoma formation are immediate effects of nuclear β-catenin translocation and WNT target gene expression. Furthermore, our inducible murine model will permit a better understanding of the molecular changes required to drive tumourigenesis in the stomach

Atomic structures of TDP-43 LCD segments and insights into reversible or pathogenic aggregation.

The normally soluble TAR DNA-binding protein 43 (TDP-43) is found aggregated both in reversible stress granules and in irreversible pathogenic amyloid. In TDP-43, the low-complexity domain (LCD) is believed to be involved in both types of aggregation. To uncover the structural origins of these two modes of β-sheet-rich aggregation, we have determined ten structures of segments of the LCD of human TDP-43. Six of these segments form steric zippers characteristic of the spines of pathogenic amyloid fibrils; four others form LARKS, the labile amyloid-like interactions characteristic of protein hydrogels and proteins found in membraneless organelles, including stress granules. Supporting a hypothetical pathway from reversible to irreversible amyloid aggregation, we found that familial ALS variants of TDP-43 convert LARKS to irreversible aggregates. Our structures suggest how TDP-43 adopts both reversible and irreversible β-sheet aggregates and the role of mutation in the possible transition of reversible to irreversible pathogenic aggregation

Pleosporales

One hundred and five generic types of Pleosporales are described and illustrated. A brief introduction and detailed history with short notes on morphology, molecular phylogeny as well as a general conclusion of each genus are provided. For those genera where the type or a representative specimen is unavailable, a brief note is given. Altogether 174 genera of Pleosporales are treated. Phaeotrichaceae as well as Kriegeriella, Zeuctomorpha and Muroia are excluded from Pleosporales. Based on the multigene phylogenetic analysis, the suborder Massarineae is emended to accommodate five families, viz. Lentitheciaceae, Massarinaceae, Montagnulaceae, Morosphaeriaceae and Trematosphaeriaceae

Cerebellar Integrity in the Amyotrophic Lateral Sclerosis - Frontotemporal Dementia Continuum

Amyotrophic lateral sclerosis (ALS) and behavioural variant frontotemporal dementia (bvFTD) are multisystem neurodegenerative disorders that manifest overlapping cognitive, neuropsychiatric and motor features. The cerebellum has long been known to be crucial for intact motor function although emerging evidence over the past decade has attributed cognitive and neuropsychiatric processes to this structure. The current study set out i) to establish the integrity of cerebellar subregions in the amyotrophic lateral sclerosis-behavioural variant frontotemporal dementia spectrum (ALS-bvFTD) and ii) determine whether specific cerebellar atrophy regions are associated with cognitive, neuropsychiatric and motor symptoms in the patients. Seventy-eight patients diagnosed with ALS, ALS-bvFTD, behavioural variant frontotemporal dementia (bvFTD), most without C9ORF72 gene abnormalities, and healthy controls were investigated. Participants underwent cognitive, neuropsychiatric and functional evaluation as well as structural imaging using voxel-based morphometry (VBM) to examine the grey matter subregions of the cerebellar lobules, vermis and crus. VBM analyses revealed: i) significant grey matter atrophy in the cerebellum across the whole ALS-bvFTD continuum; ii) atrophy predominantly of the superior cerebellum and crus in bvFTD patients, atrophy of the inferior cerebellum and vermis in ALS patients, while ALS-bvFTD patients had both patterns of atrophy. Post-hoc covariance analyses revealed that cognitive and neuropsychiatric symptoms were particularly associated with atrophy of the crus and superior lobule, while motor symptoms were more associated with atrophy of the inferior lobules. Taken together, these findings indicate an important role of the cerebellum in the ALS-bvFTD disease spectrum, with all three clinical phenotypes demonstrating specific patterns of subregional atrophy that associated with different symptomology

Observation of associated near-side and away-side long-range correlations in √sNN=5.02 TeV proton-lead collisions with the ATLAS detector

Two-particle correlations in relative azimuthal angle (Δϕ) and pseudorapidity (Δη) are measured in √sNN=5.02  TeV p+Pb collisions using the ATLAS detector at the LHC. The measurements are performed using approximately 1  μb-1 of data as a function of transverse momentum (pT) and the transverse energy (ΣETPb) summed over 3.1<η<4.9 in the direction of the Pb beam. The correlation function, constructed from charged particles, exhibits a long-range (2<|Δη|<5) “near-side” (Δϕ∼0) correlation that grows rapidly with increasing ΣETPb. A long-range “away-side” (Δϕ∼π) correlation, obtained by subtracting the expected contributions from recoiling dijets and other sources estimated using events with small ΣETPb, is found to match the near-side correlation in magnitude, shape (in Δη and Δϕ) and ΣETPb dependence. The resultant Δϕ correlation is approximately symmetric about π/2, and is consistent with a dominant cos⁡2Δϕ modulation for all ΣETPb ranges and particle pT

Search for direct pair production of the top squark in all-hadronic final states in proton-proton collisions at s√=8 TeV with the ATLAS detector

The results of a search for direct pair production of the scalar partner to the top quark using an integrated luminosity of 20.1fb−1 of proton–proton collision data at √s = 8 TeV recorded with the ATLAS detector at the LHC are reported. The top squark is assumed to decay via t˜→tχ˜01 or t˜→ bχ˜±1 →bW(∗)χ˜01 , where χ˜01 (χ˜±1 ) denotes the lightest neutralino (chargino) in supersymmetric models. The search targets a fully-hadronic final state in events with four or more jets and large missing transverse momentum. No significant excess over the Standard Model background prediction is observed, and exclusion limits are reported in terms of the top squark and neutralino masses and as a function of the branching fraction of t˜ → tχ˜01 . For a branching fraction of 100%, top squark masses in the range 270–645 GeV are excluded for χ˜01 masses below 30 GeV. For a branching fraction of 50% to either t˜ → tχ˜01 or t˜ → bχ˜±1 , and assuming the χ˜±1 mass to be twice the χ˜01 mass, top squark masses in the range 250–550 GeV are excluded for χ˜01 masses below 60 GeV

Model-based contextual policy search for data-efficient generalization of robot skills

In robotics, lower-level controllers are typically used to make the robot solve a specific task in a fixed context. For example, the lower-level controller can encode a hitting movement while the context defines the target coordinates to hit. However, in many learning problems the context may change between task executions. To adapt the policy to a new context, we utilize a hierarchical approach by learning an upper-level policy that generalizes the lower-level controllers to new contexts. A common approach to learn such upper-level policies is to use policy search. However, the majority of current contextual policy search approaches are model-free and require a high number of interactions with the robot and its environment. Model-based approaches are known to significantly reduce the amount of robot experiments, however, current model-based techniques cannot be applied straightforwardly to the problem of learning contextual upper-level policies. They rely on specific parametrizations of the policy and the reward function, which are often unrealistic in the contextual policy search formulation. In this paper, we propose a novel model-based contextual policy search algorithm that is able to generalize lower-level controllers, and is data-efficient. Our approach is based on learned probabilistic forward models and information theoretic policy search. Unlike current algorithms, our method does not require any assumption on the parametrization of the policy or the reward function. We show on complex simulated robotic tasks and in a real robot experiment that the proposed learning framework speeds up the learning process by up to two orders of magnitude in comparison to existing methods, while learning high quality policies