Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension

Bounds on the smallest eigenvalue of the neural tangent kernel (NTK) are a key ingredient in the analysis of neural network optimization and memorization. However, existing results require distributional assumptions on the data and are limited to a high-dimensional setting, where the input dimension $d_0$ scales at least logarithmically in the number of samples $n$. In this work we remove both of these requirements and instead provide bounds in terms of a measure of the collinearity of the data: notably these bounds hold with high probability even when $d_0$ is held constant versus $n$. We prove our results through a novel application of the hemisphere transform.Comment: 47 page

FoSR: First-order spectral rewiring for addressing oversquashing in GNNs

Graph neural networks (GNNs) are able to leverage the structure of graph data by passing messages along the edges of the graph. While this allows GNNs to learn features depending on the graph structure, for certain graph topologies it leads to inefficient information propagation and a problem known as oversquashing. This has recently been linked with the curvature and spectral gap of the graph. On the other hand, adding edges to the message-passing graph can lead to increasingly similar node representations and a problem known as oversmoothing. We propose a computationally efficient algorithm that prevents oversquashing by systematically adding edges to the graph based on spectral expansion. We combine this with a relational architecture, which lets the GNN preserve the original graph structure and provably prevents oversmoothing. We find experimentally that our algorithm outperforms existing graph rewiring methods in several graph classification tasks.Comment: 21 pages, accepted to ICLR 202

Optimization and Generalization of Neural Networks in Moderate-Dimensional Regimes

Deep neural networks are are able to optimize and generalize better than classical learning theory would suggest. In the past several years, this overperformance has been linked to overparameterization: learning is easier when the loss landscape is high-dimensional. However, much of the existing theory on overparameterization lives in scaling regimes that do not reflect practice, such as ones with extremely large input dimension or level of overparameterization. In this dissertation, we bridge this gap by investigating optimization and generalization in moderate-dimensional regimes of data dimension and model parameterization.In Chapter 2, we study the loss landscape of shallow and deep, mildly overparameterized ReLU neural networks on a generic dataset under the squared loss. We show by count and volume that most activation patterns correspond to parameter regions with no bad local minima. To make these results quantitative, a natural framework is to consider the smallest eigenvalue of the neural tangent kernel (NTK). Existing bounds for the smallest eigenvalue of the NTK require distributional assumptions on the data and are limited to high-dimensional input data. In Chapter 3, we bound the smallest eigenvalue of the NTK while removing both of these requirements. In Chapter 4, we investigate the generalization of neural networks with moderate-dimensional input data trained with gradient descent. We consider noisy input data that can be decomposed into a signal and a noise component, and characterize the generalization error of leaky ReLU networks in terms of the signal-to-noise ratio of the data.The core of this thesis develops theoretical frameworks capable of understanding the behavior of neural networks in moderate dimensions. This bridges the gap between existing theory in high dimensions and the scaling used in practice. This is a necessary step towards building a theory which can predict the performance of models over different choices of hyperparameters beyond limiting cases.In the remainder of this thesis, we apply ideas from learning theory to graph neural networks (GNNs). In Chapter 5, we study the phenomenon of oversquashing in graph neural networks, and frame it in terms of spectral properties of the underlying graph. In Chapter 6, we propose an algorithm drawing on this perspective which optimizes the connectivity of GNNs to improve optimization

Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape

We study the loss landscape of both shallow and deep, mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank Jacobian to many regions having deficient rank depending on the amount of overparameterization.Comment: 40 page

Myocardial GRK2 Reduces Fatty Acid Metabolism and β-Adrenergic Receptor-Mediated Mitochondrial Responses

G-protein coupled receptor (GPCR) kinase 2 (GRK2) is upregulated in heart failure (HF) patients and mouse models of cardiac disease. GRK2 is a regulator of β-adrenergic receptors (βARs), a GPCR involved in ionotropic and chronotropic responses. We and others have recently reported GRK2 to be localized in the mitochondria, although its function in the mitochondria and/or metabolism remain not clearly defined. We hypothesized that upregulation of GRK2 reduced mitochondrial respiratory function and responses to βAR activation. Utilizing isolated mouse primary adult cardiomyocytes (ACMs), we investigated the role of glucose, palmitate, ketone bodies, and BCAAs in mediating cell survival. Our results showed that myocyte upregulation of GRK2 promotes palmitate-induced cell death. Isotopologue labeling and mass spectrometry showed that the upregulation of GRK2 reduces β-hydroxybutyryl CoA generation. Next, using isoproterenol (ISO), a non-selective βAR-agonist, we determined mitochondrial function in mouse and human primary ACMs. Upregulation of GRK2 impaired ISO-mediated mitochondrial functional responses, which we propose is important for metabolic adaptations in pathological conditions. Increased cardiac levels of GRK2 reduced fatty acid-specific catabolic pathways and impaired ISO-stimulated mitochondrial function. Our data support the notion that GRK2 participates in bioenergetic remodeling and may be an important avenue for the development of novel pharmacological strategies in HF.Lewis Katz School of MedicineCardiovascular SciencesMedicineSurgeryCancer and Cellular Biolog

NEU Insights and Development of Potential Therapeutics for Pulmonary Fibrosis

Fibrosing diseases involve the formation of inappropriate scar tissue, but what drives fibrosis is unclear. Idiopathic pulmonary fibrosis involves the formation of excess scar tissue in the lungs. Sialidases (also called neuraminidases) are enzymes that desialylate glycoconjugates by cleaving terminal sialic acids from the glycoconjugates. Our lab previously found that a sialylated serum glycoprotein called serum amyloid P (SAP) inhibits fibrosis, and that sialidases attenuate SAP function. Mammals have four different sialidases, NEU1 – 4. In this dissertation, I show that extensive desialylation of glycoconjugates and upregulation of the sialidase NEU3 is observed in the fibrotic lesions in human and mouse lungs. NEU3 is upregulated in the bronchoalveolar lavage (BAL) fluid of the fibrotic mouse lungs in bleomycin-induced pulmonary fibrosis mouse model studies. Fibrosis-associated signals such as transforming growth factor-β1 (TGF-β1) and interleukin (IL)-6 upregulate NEU3 in a variety of human lung cells. Conversely, recombinant human NEU3 upregulates extracellular accumulation of active TGF-β1 and upregulates IL-6 in human peripheral blood mononuclear cells (PBMC). NEU3 also desialylates the human latency associated glyco-peptide (LAP) protein, which holds TGF-β1 in an inactive state, releasing active TGF-β1. Compared to wild-type mice, mice lacking NEU3 have significantly less bleomycin-induced pulmonary fibrosis, reduced TGF-β1 staining in the lungs after bleomycin-assault, and reduced protein, cells, and IL-6 levels in the lung fluid, providing genetic evidence for the role of NEU3 in pulmonary fibrosis in mice. Two small molecule sialidase inhibitors, DANA and oseltamivir (Tamiflu), work well on viral and mouse sialidases, but relatively poorly on human sialidases. In the bleomycin-induced pulmonary fibrosis mouse model, daily intraperitoneal injections of either DANA or oseltamivir at 10 mg / kg, starting at day 10 after bleomycin insult, strongly attenuated pulmonary fibrosis at day 21. The currently studied NEU3 inhibitors have relatively poor efficacies; thus, we designed a new class of small molecule NEU3 inhibitors. Some of our designed inhibitors have nanomolar IC50 values for the inhibition of recombinant human NEU3 releasing active TGF-β1 from the recombinant human latent TGF-β1, and inhibition of the NUE3-induced accumulation of IL-6 in human PBMC. One of our small molecule inhibitors given as daily 0.1 mg/kg injections, and two inhibitors as daily 1 mg/ kg injections, starting at day 10 after bleomycin insult, strongly attenuated pulmonary fibrosis at day 21 in the bleomycin-induced fibrosis mouse model. All of these results suggest that a NEU3-to-fibrosis-to-NEU3 positive feedback loop helps to potentiate pulmonary fibrosis. NEU3 could be a suitable target to develop treatments for lung fibrosis and our NEU3 inhibitors might be effective as therapeutics for fibrosing diseases

Sum index and difference index of graphs

Let $G$ be a nonempty simple graph with a vertex set $V(G)$ and an edge set $E(G)$. For every injective vertex labeling $f:V(G)\to\mathbb{Z}$, there are two induced edge labelings, namely $f^+:E(G)\to\mathbb{Z}$ defined by $f^+(uv)=f(u)+f(v)$, and $f^-:E(G)\to\mathbb{Z}$ defined by $f^-(uv)=|f(u)-f(v)|$. The sum index and the difference index are the minimum cardinalities of the ranges of $f^+$ and $f^-$, respectively. We provide upper and lower bounds on the sum index and difference index, and determine the sum index and difference index of various families of graphs. We also provide an interesting conjecture relating the sum index and the difference index of graphs

Methanogens, sulphate and heavy metals: a complex system

Anaerobic digestion (AD) is a well-established technology used for the treatment of wastes and wastewaters with high organic content. During AD organic matter is converted stepwise to methane-containing biogasa renewable energy carrier. Methane production occurs in the last AD step and relies on methanogens, which are rather sensitive to some contaminants commonly found in wastewaters (e.g. heavy metals), or easily outcompeted by other groups of microorganisms (e.g. sulphate reducing bacteria, SRB). This review gives an overview of previous research and pilot-scale studies that shed some light on the effects of sulphate and heavy metals on methanogenesis. Despite the numerous studies on this subject, comparison is not always possible due to differences in the experimental conditions used and parameters explained. An overview of the possible benefits of methanogens and SRB co-habitation is also covered. Small amounts of sulphide produced by SRB can precipitate with metals, neutralising the negative effects of sulphide accumulation and free heavy metals on methanogenesis. Knowledge on how to untangle and balance sulphate reduction and methanogenesis is crucial to take advantage of the potential for the utilisation of biogenic sulphide as a metal detoxification agent with minimal loss in methane production in anaerobic digesters.The research was financially supported by the People Program (Marie Curie Actions) of the European Union's Seventh Framework Programme FP7/2007-2013 under REA agreement 289193

Prostate cancer and Hedgehog signalling pathway

[Abstract] The Hedgehog (Hh) family of intercellular signalling proteins have come to be recognised as key mediators in many fundamental processes in embryonic development. Their activities are central to the growth, patterning and morphogenesis of many different regions within the bodies of vertebrates. In some contexts, Hh signals act as morphogens in the dose-dependent induction of distinct cell fates within a target field, in others as mitogens in the regulation of cell proliferation or as inducing factors controlling the form of a developing organ. These diverse functions of Hh proteins raise many intriguing questions about their mode of action. Various studies have now demonstrated the function of Hh signalling in the control of cell proliferation, especially for stem cells and stem-like progenitors. Abnormal activation of the Hh pathway has been demonstrated in a variety of human tumours. Hh pathway activity in these tumours is required for cancer cell proliferation and tumour growth. Recent studies have uncovered the role for Hh signalling in advanced prostate cancer and demonstrated that autocrine signalling by tumour cells is required for proliferation, viability and invasive behaviour. Thus, Hh signalling represents a novel pathway in prostate cancer that offers opportunities for prognostic biomarker development, drug targeting and therapeutic response monitoring

Hedgehog Signaling Regulates the Survival of Gastric Cancer Cells by Regulating the Expression of Bcl-2

Gastric cancer is the second most common cause of cancer deaths worldwide. The underlying molecular mechanisms of its carcinogenesis are relatively poorly characterized. Hedgehog (Hh) signaling, which is critical for development of various organs including the gastrointestinal tract, has been associated with gastric cancer. The present study was undertaken to reveal the underlying mechanism by which Hh signaling controls gastric cancer cell proliferation. Treatment of gastric cancer cells with cyclopamine, a specific inhibitor of Hh signaling pathway, reduced proliferation and induced apoptosis of gastric cancer cells. Cyclopamine treatment induced cytochrome c release from mitochondria and cleavage of caspase 9. Moreover, Bcl-2 expression was significantly reduced by cyclopamine treatment. These results suggest that Hh signaling regulates the survival of gastric cancer cells by regulating the expression of Bcl-2