Making Neural Networks Confidence-Calibrated and Practical

Neural networks (NNs) have become powerful tools due to their predictive accuracy. However, NNs' real-world applicability depends on accuracy and the alignment between confidence and accuracy, known as confidence calibration. Bayesian NNs (BNNs) and NN ensembles achieve good confidence calibration but are computationally expensive. In contrast, pointwise NNs are computationally efficient but poorly calibrated. Addressing these issues, this thesis proposes methods to enhance confidence calibration while maintaining or improving computational efficiency. For users preferring pointwise NNs, we propose methodology for regularising the NNs' training by using single or multiple artificial noises to improve confidence calibration and accuracy relative to standard training up to 12% without additional operations at runtime. For users able to modify the NN architecture, we propose the Single Architecture Ensemble (SAE) framework, which generalises multi-input and multi-exit architectures to embed multiple predictors into a single NN, emulating an ensemble, maintaining or improving confidence calibration and accuracy while reducing the number of compute operations or parameters by 1.5 to 3.7 times. For users who already trained an NN ensemble, we propose knowledge distillation to transfer the ensemble's predictive distribution to a single NN, marginally improving confidence calibration and accuracy, while halving the number of parameters or compute operations. We proposed uniform quantisation for BNNs, and benchmarked its impact on confidence calibration of pointwise NNs and BNNs, showing that e.g. 8-bit quantisation does not harm confidence calibration, but it reduces the memory footprint by 4 times in comparison to 32-bit floating-point precision. Lastly, we proposed an optimisation framework and a Dropout block to enable BNNs on existing field-programmable gate array-based accelerators, improving their inference latency or energy efficiency 2 to 100 times and algorithmic performance across tasks. This thesis presents methods to reduce NNs' computational costs while maintaining or improving their algorithmic performance, making confidence-calibrated NNs practical in real-world applications

MIMMO: Multi-Input Massive Multi-Output Neural Network

Neural networks (NNs) have achieved superhuman accuracy in multiple tasks, but NNs predictions' certainty is often debatable, especially if confronted with out of training distribution data. Averaging predictions of an ensemble of NNs can recalibrate the certainty of the predictions, but an ensemble is computationally expensive to deploy in practice. Recently, a new hardware-efficient multi-input multi-output (MIMO) NN was proposed to fit an ensemble of independent NNs into a single NN. In this work, we propose the addition of early-exits to the MIMO architecture with inferred depth-wise weightings to produce multiple predictions for the same input, giving a more diverse ensemble. We denote this combination as MIMMO: a multi-input, massive multi-output NN and we show that it can achieve better accuracy and calibration compared to the MIMO NN, simultaneously fit more NNs and be similarly hardware efficient as MIMO or the early-exit ensemble

Severe zinc depletion of escherichia coli: roles for high affinity zinc binding by ZinT, zinc transport and zinc-independent proteins

Zinc ions play indispensable roles in biological chemistry. However, bacteria have an impressive ability to acquire Zn2+ from the environment, making it exceptionally difficult to achieve Zn2+ deficiency, and so a comprehensive understanding of the importance of Zn2+ has not been attained. Reduction of the Zn2+ content of Escherichia coli growth medium to 60 nM or less is reported here for the first time, without recourse to chelators of poor specificity. Cells grown in Zn2+-deficient medium had a reduced growth rate and contained up to five times less cellular Zn2+. To understand global responses to Zn2+ deficiency, microarray analysis was conducted of cells grown under Zn2+-replete and Zn2+-depleted conditions in chemostat cultures. Nine genes were up-regulated more than 2-fold (p<0.05) in cells from Zn2+-deficient chemostats, including zinT (yodA). zinT is shown to be regulated by Zur ( zinc uptake regulator). A mutant lacking zinT displayed a growth defect and a 3-fold lowered cellular Zn2+ level under Zn2+ limitation. The purified ZinT protein possessed a single, high affinity metal-binding site that can accommodate Zn2+ or Cd2+. A further up-regulated gene, ykgM, is believed to encode a non-Zn2+ finger-containing paralogue of the Zn2+ finger ribosomal protein L31. The gene encoding the periplasmic Zn2+- binding protein znuA showed increased expression. During both batch and chemostat growth, cells "found" more Zn2+ than was originally added to the culture, presumably because of leaching from the culture vessel. Zn2+ elimination is shown to be a more precise method of depleting Zn2+ than by using the chelator N,N,N',N'-tetrakis(2-pyridylmethyl)ethylenediamine

Design of Single-Cylinder Internal Combustion Engine

Import 23/07/2015FERIANC, V. Návrh jednovalcového piestového spaľovacieho motora : Diplomová práca. Ostrava : VŠB – Technická univerzita Ostrava, Fakulta strojní, Institut dopravy, 2015, 86s. Vedúci práce : Ing. Dresler, P. Diplomová práca sa zaoberá návrhom spaľovacieho motora pre vozidlo Formula SAE. V úvodnej časti je popísaná funkcia, princíp činnosti a konštrukcia piestových motorov s vnútorným spaľovaním. Na základe získaných znalostí je navrhnutý prvý experimentálny model motora. Cieľom je upraviť model tak, aby vyhovoval stanoveným požiadavkám a špecifikáciám súťaže. Ďalším cieľom je na základe získaných dát navrhnúť konštrukčné riešenie vo virtuálnom prostredí a u vybraného celku spraviť pevnostnú kontrolu.FERIANC, V. Design of Single-Cylinder Internal Combustion Engine : Master Thesis. Ostrava: VŠB – Technical University of Ostrava, Faculty of Mechanical Engineering, Institute of Transport, 2015, 86 pages. Thesis head: Ing. Dresler, P. This Master thesis deals with the design of combustion engine of the vehicle, Formula SAE. In introduction we described the function, working principle and construction of piston engines with internal combustion. On the base of acquired knowledge there is designed the first experimental model of the engine. The aim is to modify the model in the way to be suitable for accepted requirements and specifications of the competition. Another aim is to design the constructional solution in virtual environment on the base of acquired data and to make a load control of selected part.342 - Institut dopravyvýborn

Simple Regularisation for Uncertainty-Aware Knowledge Distillation

Considering uncertainty estimation of modern neural networks (NNs) is one of the most important steps towards deploying machine learning systems to meaningful real-world applications such as in medicine, finance or autonomous systems. At the moment, ensembles of different NNs constitute the state-of-the-art in both accuracy and uncertainty estimation in different tasks. However, ensembles of NNs are unpractical under real-world constraints, since their computation and memory consumption scale linearly with the size of the ensemble, which increase their latency and deployment cost. In this work, we examine a simple regularisation approach for distribution-free knowledge distillation of ensemble of machine learning models into a single NN. The aim of the regularisation is to preserve the diversity, accuracy and uncertainty estimation characteristics of the original ensemble without any intricacies, such as fine-tuning. We demonstrate the generality of the approach on combinations of toy data, SVHN/CIFAR-10, simple to complex NN architectures and different tasks

ComBiNet: Compact Convolutional Bayesian Neural Network for Image Segmentation

Fully convolutional U-shaped neural networks have largely been the dominant approach for pixel-wise image segmentation. In this work, we tackle two defects that hinder their deployment in real-world applications: 1) Predictions lack uncertainty quantification that may be crucial to many decision-making systems; 2) Large memory storage and computational consumption demanding extensive hardware resources. To address these issues and improve their practicality we demonstrate a few-parameter compact Bayesian convolutional architecture, that achieves a marginal improvement in accuracy in comparison to related work using significantly fewer parameters and compute operations. The architecture combines parameter-efficient operations such as separable convolutions, bilinear interpolation, multi-scale feature propagation and Bayesian inference for per-pixel uncertainty quantification through Monte Carlo Dropout. The best performing configurations required fewer than 2.5 million parameters on diverse challenging datasets with few observations

Learning-based MPC with uncertainty estimation for resilient microgrid energy management

To enhance fault resilience in microgrid systems at the energy management level, this paper introduces a novel proactive scheduling algorithm, based on uncertainty modelling thanks to a specifically designed neural network. The algorithm is trained and deployed online and it estimates uncertainties in predicting future load demands and other relevant profiles. We integrate the novel learning algorithm with a stochastic model predictive control, enabling the microgrid to store sufficient energy to adaptively deal with possible faults. Experimental results show that a reliable estimation of the unknown profiles' mean and variance is obtained, improving the robustness of proactive scheduling strategies against uncertainties

An Online Learning Method for Microgrid Energy Management Control*

We propose a novel Model Predictive Control (MPC) scheme based on online-learning (OL) for microgrid energy management, where the control optimisation is embedded as the last layer of the neural network. The proposed MPC scheme deals with uncertainty on the load and renewable generation power profiles and on electricity prices, by employing the predictions provided by an online trained neural network in the optimisation problem. In order to adapt to possible changes in the environment, the neural network is online trained based on continuously received data. The network hyperparameters are selected by performing a hyperparameter optimisation before the deployment of the controller, using a pretraining dataset. We show the effectiveness of the proposed method for microgrid energy management through extensive experiments on real microgrid datasets. Moreover, we show that the proposed algorithm has good transfer learning (TL) capabilities among different microgrids

Navigating Noise: A Study of How Noise Influences Generalisation and Calibration of Neural Networks

Enhancing the generalisation abilities of neural networks (NNs) through integrating noise such as MixUp or Dropout during training has emerged as a powerful and adaptable technique. Despite the proven efficacy of noise in NN training, there is no consensus regarding which noise sources, types and placements yield maximal benefits in generalisation and confidence calibration. This study thoroughly explores diverse noise modalities to evaluate their impacts on NN's generalisation and calibration under in-distribution or out-of-distribution settings, paired with experiments investigating the metric landscapes of the learnt representations across a spectrum of NN architectures, tasks, and datasets. Our study shows that AugMix and weak augmentation exhibit cross-task effectiveness in computer vision, emphasising the need to tailor noise to specific domains. Our findings emphasise the efficacy of combining noises and successful hyperparameter transfer within a single domain but the difficulties in transferring the benefits to other domains. Furthermore, the study underscores the complexity of simultaneously optimising for both generalisation and calibration, emphasising the need for practitioners to carefully consider noise combinations and hyperparameter tuning for optimal performance in specific tasks and datasets.Comment: Accepted at Transactions on Machine Learning Research (April 2024). Martin and Ondrej contributed equall