Taming False Positives in Out-of-Distribution Detection with Human Feedback

Robustness to out-of-distribution (OOD) samples is crucial for safely deploying machine learning models in the open world. Recent works have focused on designing scoring functions to quantify OOD uncertainty. Setting appropriate thresholds for these scoring functions for OOD detection is challenging as OOD samples are often unavailable up front. Typically, thresholds are set to achieve a desired true positive rate (TPR), e.g., $95\%$ TPR. However, this can lead to very high false positive rates (FPR), ranging from 60 to 96\%, as observed in the Open-OOD benchmark. In safety-critical real-life applications, e.g., medical diagnosis, controlling the FPR is essential when dealing with various OOD samples dynamically. To address these challenges, we propose a mathematically grounded OOD detection framework that leverages expert feedback to \emph{safely} update the threshold on the fly. We provide theoretical results showing that it is guaranteed to meet the FPR constraint at all times while minimizing the use of human feedback. Another key feature of our framework is that it can work with any scoring function for OOD uncertainty quantification. Empirical evaluation of our system on synthetic and benchmark OOD datasets shows that our method can maintain FPR at most $5\%$ while maximizing TPR.Comment: Appeared in the 27th International Conference on Artificial Intelligence and Statistics (AISTATS 2024

Quantum criticality driven by the cavity coupling in Rabi-dimer model

The superradiant phase transition (SPT) controlled by the interacting strength between the two-level atom and the photons has been a hot topic in the Rabi model and the Rabi-dimer model. The latter describes two Rabi cavities coupled with an inter-cavity hopping parameter. Moreover, the SPT in the Rabi-dimer model is found to be the same universal class that in the Rabi model by investigating the correlation-length critical exponent. In this paper, we are concerned about whether the inter-cavity hopping parameter between two Rabi cavities (i.e., the Rabi-dimer model) will induce the SPT and to which the universal class of the phase transition belongs. We analytically derive the phase boundary of the SPT and investigate the ground-state properties of the system. We uncover that the inter-cavity induced SPT can be apparently understood from the ground-state energy and the ground-state photon population, as well as the ground-state expectation value of the squared anti-symmetric mode. From the scaling analysis of the fidelity susceptibility, we numerically verify that the SPT driven by the cavity coupling belongs to the same universal class as the one driven by the atom-cavity interaction. Our work enriches the studies on the SPT and its critical behaviors in the Rabi-dimer model.Comment: 11 pages, 6 figure

Good Data from Bad Models : Foundations of Threshold-based Auto-labeling

Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Auto-labeling systems are a promising way to reduce reliance on manual labeling for dataset construction. Threshold-based auto-labeling, where validation data obtained from humans is used to find a threshold for confidence above which the data is machine-labeled, is emerging as a popular solution used widely in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. In this work, we analyze threshold-based auto-labeling systems and derive sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of machine-labeled data. Our results provide two insights. First, reasonable chunks of the unlabeled data can be automatically and accurately labeled by seemingly bad models. Second, a hidden downside of threshold-based auto-labeling systems is potentially prohibitive validation data usage. Together, these insights describe the promise and pitfalls of using such systems. We validate our theoretical guarantees with simulations and study the efficacy of threshold-based auto-labeling on real datasets

Detecting Attacks in CyberManufacturing Systems: Additive Manufacturing Example

CyberManufacturing System is a vision for future manufacturing where physical components are fully integrated with computational processes in a connected environment. However, realizing the vision requires that its security be adequately ensured. This paper presents a vision-based system to detect intentional attacks on additive manufacturing processes, utilizing machine learning techniques. Particularly, additive manufacturing systems have unique vulnerabilities to malicious attacks, which can result in defective infills but without affecting the exterior. In order to detect such infill defects, the research uses simulated 3D printing process images as well as actual 3D printing process images to compare accuracies of machine learning algorithms in classifying, clustering and detecting anomalies on different types of infills. Three algorithms - (i) random forest, (ii) k nearest neighbor, and (iii) anomaly detection - have been adopted in the research and shown to be effective in detecting such defects

The relations between metabolic variations and genetic evolution of different species

Metabonomics has been applied in many bio-related scientific fields. Nevertheless, some animal research works are shown to fail when they are extended to humans. Therefore, it is essential to figure out suitable animal modeling to mimic human metabolism so that animal findings can serve humans. In this study, two kinds of commonly selected body fluids, serum and urine, from humans and various experimental animals were characterized by integration of nuclear magnetic resonance (NMR) spectroscopy with multivariate statistical analysis to identify the interspecies metabolic differences and similarities at a baseline physiological status. Our results highlight that the dairy cow and pig may be an optimal choice for transportation and biodistribution studies of drugs and that the Kunming (KM) mouse model may be the most effective for excretion studies of drugs, whereas the Sprague-Dawley (SD) rat could be the most suitable candidate for animal modeling under overall considerations. The biochemical pathways analyses further provide an interconnection between genetic evolution and metabolic variations, where species evolution most strongly affects microbial biodiversity and, consequently, has effects on the species-specific biological substances of biosynthesis and corresponding biological activities. Knowledge of the metabolic effects from species difference will enable the construction of better models for disease diagnosis, drug metabolism, and toxicology research. (C) 2015 Elsevier Inc. All rights reserved.Metabonomics has been applied in many bio-related scientific fields. Nevertheless, some animal research works are shown to fail when they are extended to humans. Therefore, it is essential to figure out suitable animal modeling to mimic human metabolism so that animal findings can serve humans. In this study, two kinds of commonly selected body fluids, serum and urine, from humans and various experimental animals were characterized by integration of nuclear magnetic resonance (NMR) spectroscopy with multivariate statistical analysis to identify the interspecies metabolic differences and similarities at a baseline physiological status. Our results highlight that the dairy cow and pig may be an optimal choice for transportation and biodistribution studies of drugs and that the Kunming (KM) mouse model may be the most effective for excretion studies of drugs, whereas the Sprague-Dawley (SD) rat could be the most suitable candidate for animal modeling under overall considerations. The biochemical pathways analyses further provide an interconnection between genetic evolution and metabolic variations, where species evolution most strongly affects microbial biodiversity and, consequently, has effects on the species-specific biological substances of biosynthesis and corresponding biological activities. Knowledge of the metabolic effects from species difference will enable the construction of better models for disease diagnosis, drug metabolism, and toxicology research. (C) 2015 Elsevier Inc. All rights reserved

Performance Evaluation of Multidimensional Poverty Alleviation In Guizhou

<p>The single currency dimension anti-poverty measures are no longer applicable to the present, and multi-dimensional poverty alleviation is an inevitable choice of history. Based on the time series data of Guizhou Province from 2015 to 2019, the performance evaluation index system of poverty alleviation is constructed based on the four dimensions of economic development level, social development level, production and life development level and ecological environment level in Guizhou Province. The entropy weight-topsis model is used to calculate the entropy value and difference coefficient, and the corresponding weight is obtained. The topsis method is used to calculate the score to obtain the evaluation of Guizhou 's multi-dimensional poverty alleviation performance. The study found that Guizhou 's multidimensional poverty alleviation has experienced a process of slowing down first, then intensifying and then slowing down. Based on this, an innovative poverty alleviation model is proposed to improve poverty alleviation performance ; improve public services and enhance the level of social development ; strengthen environmental awareness and promote the construction of ecological poverty alleviation ; strengthen environmental awareness, promote ecological poverty alleviation and other measures.</p&gt