Advancing Efficiency in Neural Networks through Sparsity and Feature Selection

Deep neural networks (DNNs) have attracted considerable attention over the last several years due to their promising results in various applications. Nevertheless, their extensive model size and over-parameterization have brought to the forefront a significant challenge—escalating computational costs. Furthermore, these challenges are exacerbated when dealing with high-dimensional data, as the complexity and resource requirements of DNNs increase significantly. Consequently, the utilization of deep learning models proves to be ill-suited for scenarios characterized by constrained computational resources and limited battery life, incurring substantial training and inference costs, both in terms of memory and computational resources.Sparse neural networks (SNNs) have emerged as a prominent approach toward addressing the over-parameterization inherent in DNNs, thus, mitigating associated costs. By keeping only the most important connections of a DNN, they achieve a comparable result to their dense counterpart network but with significantly fewer parameters. However, most current solutions to reduce computation costs using SNNs mainly gain inference efficiency, while being resource-intensive during training. Furthermore, these solutions predominantly center their efforts on a restricted set of application domains, particularly within the realms of vision and language tasks.This Ph.D. research aims to address these challenges by introducing Cost-effective Artificial Neural Networks (CeANNs) designed to achieve a targeted performance across diverse complex machine learning tasks while demanding minimal computational, memory, and energy resources during both network training and inference. Our study on CeANNs includes two primary perspectives: model and data efficiency. In essence, we leverage the potential of SNNs to reduce the model parameters and data dimensionality, thereby facilitating efficient training and deployment of artificial neural networks. This work results in the development of artificial neural networks that are more practical and accessible for real-world applications, with a key emphasis on cost-effectiveness. Within this thesis, we delve into our developed methodologies aimed at advancing efficiency. Our contributions can be summarized as follows:Part I. Advancing Training and Inference Efficiency of DNNs through Sparsity.This part of the thesis focuses on enhancing the model efficiency of DNNs through sparsity. The inherent high computational cost associated with DNNs, primarily stemming from their large, over-parameterized layers, highlights the need for computationally-aware design in both model architecture and training methods. Within Part I of this thesis, we leverage sparsity to address this challenge, with a specific focus on achieving a targeted performance in extremely sparse neural networks and efficient time series analysis with DNNs. We propose two algorithms to tackle these issues: a dynamic sparse training (DST) algorithm for learning in extremely sparse neural networks (Chapter 2) and a methodology for obtaining SNNs for time series prediction (Chapter 3). In essence, our goal is to enhance the training and inference efficiency of DNNs through sparsity while focusing on addressing specific challenges in underexplored application domains, particularly in tabular and time series data analysis.Part II. Leveraging Feature Selection for Efficient Model Development. In the pursuit of cost-effective artificial neural networks, it is crucial to address the challenges associated with high-dimensional input data due to its potential to hinder scalability and introduce issues such as the curse of dimensionality and over-fitting. One promising avenue to tackle these challenges is feature selection, a technique designed to identify the most relevant and informative attributes of a dataset. However, existing feature selection methods are mostly computationally expensive, especially when dealing with high-dimensional datasets or those with a substantial sample size. To address this issue, in the second part of the thesis, we propose for the first time to exploit SNNs to perform efficient feature selection. We present our two proposed feature selection methods, one for unsupervised feature selection (Chapter 4) and another for supervised feature selection (Chapter 5). These methods are specifically designed to offer effective solutions to the challenges of high dimensionality while maintaining computational efficiency. As we show in Chapter 5, by using less than $10\%$ of the parameters of the dense network, our proposed method achieves the highest ranking-based score in terms of finding qualitative features among the state-of-the-art feature selection methods. The combination of feature selection and neural networks offers a powerful strategy, enhancing the training process and performing dimensionality reduction, thereby advancing the overall efficiency of model development.In conclusion, this research focuses on the development of cost-effective artificial neural networks that deliver targeted performance while minimizing computational, memory, and energy resources. The research explores CeANNs from two perspectives: model efficiency and data efficiency. The first part of the thesis addresses model efficiency through sparsity, proposing algorithms for efficient training and inference of DNNs for various data types. The second part of the thesis leverages SNNs to efficiently select an informative subset of attributes from high-dimensional input data. By considering both model and data efficiency, the aim is to develop CeANNs that are practical and accessible for real-world applications. In Chapter 6, we present the preliminary impact and the limitations of the work and potential directions for future research in the field. We hope that this Ph.D. thesis will pave the way to designing cost-effective artificial neural networks

Трендсеттінг як ключовий фактор управління інноваційними ризиками індустрії моди

Індустрія моди нового тисячоріччя перетворилася в багатомільйонний сектор економіки, у котрому інноваційна діяльність грає ключову роль. Інновації в дизайні сучасного костюма з інструмента вдосконалювання характеристик об’єкта перетворюються в одну з основних його характеристик, тому фешн-проекти є інноваційними за своєю природою [2]

Benzene degradation in a denitrifying biofilm reactor: activity and microbial community composition

Benzene is an aromatic compound and harmful for the environment. Biodegradation of benzene can reduce the toxicological risk after accidental or controlled release of this chemical in the environment. In this study, we further characterized an anaerobic continuous biofilm culture grown for more than 14 years on benzene with nitrate as electron acceptor. We determined steady state degradation rates, microbial community composition dynamics in the biofilm, and the initial anaerobic benzene degradation reactions. Benzene was degraded at a rate of 0.15 μmol/mg protein/day and a first-order rate constant of 3.04/day which was fourfold higher than rates reported previously. Bacteria belonging to the Peptococcaceae were found to play an important role in this anaerobic benzene-degrading biofilm culture, but also members of the Anaerolineaceae were predicted to be involved in benzene degradation or benzene metabolite degradation based on Illumina MiSeq analysis of 16S ribosomal RNA genes. Biomass retention in the reactor using a filtration finger resulted in reduction of benzene degradation capacity. Detection of the benzene carboxylase encoding gene, abcA, and benzoic acid in the culture vessel indicated that benzene degradation proceeds through an initial carboxylation step.</p

Prospects for harnessing biocide resistance for bioremediation and detoxification

Prokaryotes in natural environments respond rapidly to high concentrations of chemicals and physical stresses. Exposure to anthropogenic toxic substancessuch as oil, chlorinated solvents, or antibioticsfavors the evolution of resistant phenotypes, some of which can use contaminants as an exclusive carbon source or as electron donors and acceptors. Microorganisms similarly adapt to extreme pH, metal, or osmotic stress. The metabolic plasticity of prokaryotes can thus be harnessed for bioremediation and can be exploited in a variety of ways, ranging from stimulated natural attenuation to bioaugmentation and from wastewater treatment to habitat restoration.We thank H. Stroo (Stroo Consulting) and C. Aziz (Ramboll) for providing photographs of bioaugmentation with OHRB, and H. Patzelt (Mazoon Environmental and Technological Services) for providing photographs of bioaugmentation with halophilic microorganisms. Funding: S.A., I.S.-A., and A.J.M.S. are supported by the Netherlands Ministry of Education, Culture and Science (project 024.002.002) and advanced ERC grant (project 323009). H.S. and S.A. were supported by a grant of BE-Basic-FES funds from the Dutch Ministry of Economic Affairs. H.S., J.R.v.d.M., and H.J.H. were supported by the European Commission (BACSIN, contract 211684; P4SB, contract 633962).info:eu-repo/semantics/publishedVersio

Geochemical and microbial community determinants of reductive dechlorination at a site biostimulated with glycerol

Biostimulation is widely used to enhance reductive dechlorination of chlorinated ethenes in contaminated aquifers. However, the knowledge on corresponding biogeochemical responses is limited. In this study glycerol was injected in an aquifer contaminated with cis-dichloroethene (cDCE), and geochemical and microbial shifts were followed for 265 days. Consistent with anoxic conditions and sulfate reduction after biostimulation, MiSeq 16S rRNA gene sequencing revealed temporarily increased relative abundance of Firmicutes, Bacteriodetes and sulfate reducing Deltaproteobacteria. In line with 13C cDCE enrichment and increased Dehalococcoides mccartyi (Dcm) numbers, dechlorination was observed towards the end of the field experiment, albeit being incomplete with accumulation of vinyl chloride. This was concurrent with i) decreased concentrations of dissolved organic carbon (DOC), reduced relative abundances of fermenting and sulfate reducing bacteria that have been suggested to promote Dcm growth by providing electron donor (H2) and essential corrinoid cofactors, ii) increased sulfate concentration and increased relative abundance of Epsilonproteobacteria and Deferribacteres as putative oxidizers of reduced sulfur compounds. Strong correlations of DOC, relative abundance of fermenters and sulfate reducers, and dechlorination imply the importance of syntrophic interactions to sustain robust dechlorination. Tracking microbial and environmental parameters that promote/preclude enhanced reductive dechlorination should aid development of sustainable bioremediation strategies. This article is protected by copyright. All rights reserved.This study was supported by a VITO/KU Leuven PhD scholarship (EU FP7 project AQUAREHAB, grant 226565) to S Atashgahi. Furthermore, S Atashgahi and H Smidt received support bya grant ofBE-Basic-FES funds from theDutch Ministry of Economic Affairs and D Springael by the InterUniversity Attraction Pole (IUAP) “m-manager” of the Belgian Science Policy (BELSPO, P7/25). We thankRichard Lookman for his assistance in the field experiment and acknowledge the China Scholarship Council for the support to Y Lu and Y Zheng.info:eu-repo/semantics/publishedVersio

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers

Efficient time series forecasting has become critical for real-world applications, particularly with deep neural networks (DNNs). Efficiency in DNNs can be achieved through sparse connectivity and reducing the model size. However, finding the sparsity level automatically during training remains a challenging task due to the heterogeneity in the loss-sparsity tradeoffs across the datasets. In this paper, we propose \enquote{\textbf{P}runing with \textbf{A}daptive \textbf{S}parsity \textbf{L}evel} (\textbf{PALS}), to automatically seek an optimal balance between loss and sparsity, all without the need for a predefined sparsity level. PALS draws inspiration from both sparse training and during-training methods. It introduces the novel "expand" mechanism in training sparse neural networks, allowing the model to dynamically shrink, expand, or remain stable to find a proper sparsity level. In this paper, we focus on achieving efficiency in transformers known for their excellent time series forecasting performance but high computational cost. Nevertheless, PALS can be applied directly to any DNN. In the scope of these arguments, we demonstrate its effectiveness also on the DLinear model. Experimental results on six benchmark datasets and five state-of-the-art transformer variants show that PALS substantially reduces model size while maintaining comparable performance to the dense model. More interestingly, PALS even outperforms the dense model, in 12 and 14 cases out of 30 cases in terms of MSE and MAE loss, respectively, while reducing 65% parameter count and 63% FLOPs on average. Our code will be publicly available upon acceptance of the paper

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers

Efficient time series forecasting has become critical for real-world applications, particularly with deep neural networks (DNNs). Efficiency in DNNs can be achieved through sparse connectivity and reducing the model size. However, finding the sparsity level automatically during training remains a challenging task due to the heterogeneity in the loss-sparsity tradeoffs across the datasets. In this paper, we propose \enquote{\textbf{P}runing with \textbf{A}daptive \textbf{S}parsity \textbf{L}evel} (\textbf{PALS}), to automatically seek an optimal balance between loss and sparsity, all without the need for a predefined sparsity level. PALS draws inspiration from both sparse training and during-training methods. It introduces the novel "expand" mechanism in training sparse neural networks, allowing the model to dynamically shrink, expand, or remain stable to find a proper sparsity level. In this paper, we focus on achieving efficiency in transformers known for their excellent time series forecasting performance but high computational cost. Nevertheless, PALS can be applied directly to any DNN. In the scope of these arguments, we demonstrate its effectiveness also on the DLinear model. Experimental results on six benchmark datasets and five state-of-the-art transformer variants show that PALS substantially reduces model size while maintaining comparable performance to the dense model. More interestingly, PALS even outperforms the dense model, in 12 and 14 cases out of 30 cases in terms of MSE and MAE loss, respectively, while reducing 65% parameter count and 63% FLOPs on average. Our code will be publicly available upon acceptance of the paper