A mathematical theory of semantic development in deep neural networks

An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: what are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep learning dynamics to give rise to these regularities

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed can nevertheless remain finite: for a special class of initial conditions on the weights, very deep networks incur only a finite, depth independent, delay in learning speed relative to shallow networks. We show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, while scaled random Gaussian initializations cannot. We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times. We further show that these initial conditions also lead to faithful propagation of gradients even in deep nonlinear networks, as long as they operate in a special regime known as the edge of chaos.Comment: Submission to ICLR2014. Revised based on reviewer feedbac

GaAs monolithic frequency doublers with series connected varactor diodes

GaAs monolithic frequency doublers using series connected varactor diodes have been fabricated for the first time. Output powers of 150 mW at 36.9 GHz with 24% efficiency and 300 mW at 24.8 GHz with 18% efficiency have been obtained. Peak efficiencies of 35% at output power levels near 100 mW have been achieved at both frequencies. Both K-band and Ka-band frequency doublers are derived from a lower power, single-diode design by series connection of two diodes and scaling to achieve different power and frequency specifications. Their fabrication was accomplished using the same process sequence

The ACIGA Data Analysis programme

The Data Analysis programme of the Australian Consortium for Interferometric Gravitational Astronomy (ACIGA) was set up in 1998 by the first author to complement the then existing ACIGA programmes working on suspension systems, lasers and optics, and detector configurations. The ACIGA Data Analysis programme continues to contribute significantly in the field; we present an overview of our activities.Comment: 10 pages, 0 figures, accepted, Classical and Quantum Gravity, (Proceedings of the 5th Edoardo Amaldi Conference on Gravitational Waves, Tirrenia, Pisa, Italy, 6-11 July 2003

Risk Factors for Long-Term Coronary Artery Calcium Progression in the Multi-Ethnic Study of Atherosclerosis.

BackgroundCoronary artery calcium (CAC) detected by noncontrast cardiac computed tomography scanning is a measure of coronary atherosclerosis burden. Increasing CAC levels have been strongly associated with increased coronary events. Prior studies of cardiovascular disease risk factors and CAC progression have been limited by short follow-up or restricted to patients with advanced disease.Methods and resultsWe examined cardiovascular disease risk factors and CAC progression in a prospective multiethnic cohort study. CAC was measured 1 to 4 times (mean 2.5 scans) over 10 years in 6810 adults without preexisting cardiovascular disease. Mean CAC progression was 23.9 Agatston units/year. An innovative application of mixed-effects models investigated associations between cardiovascular disease risk factors and CAC progression. This approach adjusted for time-varying factors, was flexible with respect to follow-up time and number of observations per participant, and allowed simultaneous control of factors associated with both baseline CAC and CAC progression. Models included age, sex, study site, scanner type, and race/ethnicity. Associations were observed between CAC progression and age (14.2 Agatston units/year per 10 years [95% CI 13.0 to 15.5]), male sex (17.8 Agatston units/year [95% CI 15.3 to 20.3]), hypertension (13.8 Agatston units/year [95% CI 11.2 to 16.5]), diabetes (31.3 Agatston units/year [95% CI 27.4 to 35.3]), and other factors.ConclusionsCAC progression analyzed over 10 years of follow-up, with a novel analytical approach, demonstrated strong relationships with risk factors for incident cardiovascular events. Longitudinal CAC progression analyzed in this framework can be used to evaluate novel cardiovascular risk factors

Performance of alkaline battery cells used in emergency locator transmitters

The characteristics of battery power supplies for emergency locator transmitters (ELT's) were investigated by testing alkaline zinc/manganese dioxide cells of the type typically used in ELT's. Cells from four manufacturers were tested. The cells were subjected to simulated environmental and load conditions representative of those required for survival and operation. Battery cell characteristics that may contribute to ELT malfunctions and limitations were evaluated. Experimental results from the battery cell study are discussed, and an evaluation of ELT performance while operating under a representative worst-case environmental condition is presented

Binary Population and Spectral Synthesis Version 2.1: construction, observational verification and new results

The Binary Population and Spectral Synthesis (BPASS) suite of binary stellar evolution models and synthetic stellar populations provides a framework for the physically motivated analysis of both the integrated light from distant stellar populations and the detailed properties of those nearby. We present a new version 2.1 data release of these models, detailing the methodology by which BPASS incorporates binary mass transfer and its effect on stellar evolution pathways, as well as the construction of simple stellar populations. We demonstrate key tests of the latest BPASS model suite demonstrating its ability to reproduce the colours and derived properties of resolved stellar populations, including well- constrained eclipsing binaries. We consider observational constraints on the ratio of massive star types and the distribution of stellar remnant masses. We describe the identification of supernova progenitors in our models, and demonstrate a good agreement to the properties of observed progenitors. We also test our models against photometric and spectroscopic observations of unresolved stellar populations, both in the local and distant Universe, finding that binary models provide a self-consistent explanation for observed galaxy properties across a broad redshift range. Finally, we carefully describe the limitations of our models, and areas where we expect to see significant improvement in future versions.Comment: 69 pages, 45 figures. Accepted for publication in PASA. Accompanied by a full, documented data release at http://bpass.auckland.ac.nz and http://warwick.ac.uk/bpas