Variational Analysis of Constrained M-Estimators

We propose a unified framework for establishing existence of nonparametric M-estimators, computing the corresponding estimates, and proving their strong consistency when the class of functions is exceptionally rich. In particular, the framework addresses situations where the class of functions is complex involving information and assumptions about shape, pointwise bounds, location of modes, height at modes, location of level-sets, values of moments, size of subgradients, continuity, distance to a "prior" function, multivariate total positivity, and any combination of the above. The class might be engineered to perform well in a specific setting even in the presence of little data. The framework views the class of functions as a subset of a particular metric space of upper semicontinuous functions under the Attouch-Wets distance. In addition to allowing a systematic treatment of numerous M-estimators, the framework yields consistency of plug-in estimators of modes of densities, maximizers of regression functions, level-sets of classifiers, and related quantities, and also enables computation by means of approximating parametric classes. We establish consistency through a one-sided law of large numbers, here extended to sieves, that relaxes assumptions of uniform laws, while ensuring global approximations even under model misspecification

Fusion of Hard and Soft Information in Nonparametric Density Estimation

This article discusses univariate density estimation in situations when the sample (hard information) is supplemented by “soft” information about the random phenomenon. These situations arise broadly in operations research and management science where practical and computational reasons severely limit the sample size, but problem structure and past experiences could be brought in. In particular, density estimation is needed for generation of input densities to simulation and stochastic optimization models, in analysis of simulation output, and when instantiating probability models. We adopt a constrained maximum likelihood estimator that incorporates any, possibly random, soft information through an arbitrary collection of constraints. We illustrate the breadth of possibilities by discussing soft information about shape, support, continuity, smoothness, slope, location of modes, symmetry, density values, neighborhood of known density, moments, and distribution functions. The maximization takes place over spaces of extended real-valued semicontinuous functions and therefore allows us to consider essentially any conceivable density as well as convenient exponential transformations. The infinite dimensionality of the optimization problem is overcome by approximating splines tailored to these spaces. To facilitate the treatment of small samples, the construction of these splines is decoupled from the sample. We discuss existence and uniqueness of the estimator, examine consistency under increasing hard and soft information, and give rates of convergence. Numerical examples illustrate the value of soft information, the ability to generate a family of diverse densities, and the effect of misspecification of soft information.U.S. Army Research Laboratory and the U.S. Army Research Office grant 00101-80683U.S. Army Research Laboratory and the U.S. Army Research Office grant W911NF-10-1-0246U.S. Army Research Laboratory and the U.S. Army Research Office grant W911NF-12-1-0273U.S. Army Research Laboratory and the U.S. Army Research Office grant 00101-80683U.S. Army Research Laboratory and the U.S. Army Research Office grant W911NF-10-1-0246U.S. Army Research Laboratory and the U.S. Army Research Office grant W911NF-12-1-027

Data-driven optimization of reliability using buffered failure probability

Design and operation of complex engineering systems rely on reliability optimization. Such optimization requires us to account for uncertainties expressed in terms of compli-cated, high-dimensional probability distributions, for which only samples or data might be available. However, using data or samples often degrades the computational efficiency, particularly as the conventional failure probability is estimated using the indicator function whose gradient is not defined at zero. To address this issue, by leveraging the buffered failure probability, the paper develops the buffered optimization and reliability method (BORM) for efficient, data-driven optimization of reliability. The proposed formulations, algo-rithms, and strategies greatly improve the computational efficiency of the optimization and thereby address the needs of high-dimensional and nonlinear problems. In addition, an analytical formula is developed to estimate the reliability sensitivity, a subject fraught with difficulty when using the conventional failure probability. The buffered failure probability is thoroughly investigated in the context of many different distributions, leading to a novel measure of tail-heaviness called the buffered tail index. The efficiency and accuracy of the proposed optimization methodology are demonstrated by three numerical examples, which underline the unique advantages of the buffered failure probability for data-driven reliability analysis

S-BORM: Reliability-based optimization of general systems using buffered optimization and reliability method

Reliability-based optimization (RBO) is crucial for identifying optimal risk-informed decisions for designing and operating engineering systems. However, its computation remains challenging as it requires a concurrent task of optimization and reliability analysis. Moreover, computation becomes even more complicated when considering performance of a general system, whose failure event is represented as a link-set of cut-sets. This is because even when component events have smooth and convex limit-state functions, the system limit-state function has neither property, except in trivial cases. To address the challenge, this study develops an efficient algorithm to solve RBO problems of general system events. We employ the buffered optimization and reliability method (BORM), which utilizes, instead of the conventional failure probability definition, the buffered failure probability. The proposed algorithm solves a sequence of difference-of-convex RBO models iteratively by employing a proximal bundle method. For demonstration, we design three numerical examples with increasing complexity that includes up to 108 cut-sets, which are solved by the proposed algorithm within a minute with high accuracy. We also demonstrate its robustness by performing extensive parametric studies

A virtual chromoendoscopy artificial intelligence system to detect endoscopic and histologic activity/remission and predict clinical outcomes in ulcerative colitis

Background Endoscopic and histological remission (ER, HR) are therapeutic targets in ulcerative colitis (UC). Virtual chromoendoscopy (VCE) improves endoscopic assessment and the prediction of histology; however, interobserver variability limits standardized endoscopic assessment. We aimed to develop an artificial intelligence (AI) tool to distinguish ER/activity, and predict histology and risk of flare from white-light endoscopy (WLE) and VCE videos. Methods 1090 endoscopic videos (67 280 frames) from 283 patients were used to develop a convolutional neural network (CNN). UC endoscopic activity was graded by experts using the Ulcerative Colitis Endoscopic Index of Severity (UCEIS) and Paddington International virtual ChromoendoScopy ScOre (PICaSSO). The CNN was trained to distinguish ER/activity on endoscopy videos, and retrained to predict HR/activity, defined according to multiple indices, and predict outcome; CNN and human agreement was measured. Results The AI system detected ER (UCEIS = 1) in WLE videos with 72% sensitivity, 87% specificity, and an area under the receiver operating characteristic curve (AUROC) of 0.85; for detection of ER in VCE videos (PICaSSO = 3), the sensitivity was 79 %, specificity 95%, and the AUROC 0.94. The prediction of HR was similar between WLE and VCE videos (accuracies ranging from 80% to 85%). The model s stratification of risk of flare was similar to that of physician-assessed endoscopy scores. Conclusions Our system accurately distinguished ER/activity and predicted HR and clinical outcome from colonoscopy videos. This is the first computer model developed to detect inflammation/healing on VCE using the PICaSSO and the first computer tool to provide endoscopic, histologic, and clinical assessment

Specifying and Validating Probabilistic Inputs for Prescriptive Models of Decision Making over Time

Optimization models for making decisions over time in uncertain environments rely on probabilistic inputs, such as scenario trees for stochastic mathematical programs. The quality of model outputs, i.e., the solutions obtained, depends on the quality of these inputs. However, solution quality is rarely assessed in a rigorous way. The connection between validation of model inputs and quality of the resulting solution is not immediate. This chapter discusses some efforts to formulate realistic probabilistic inputs and subsequently validate them in terms of the quality of solutions they produce. These include formulating probabilistic models based on statistical descriptions understandable to decision makers; conducting statistical tests to assess the validity of stochastic process models and their discretization; and conducting re-enactments to assess the quality of the formulation in terms of solution performance against observational data. Studies of long-term capacity expansion in service industries, including electric power, and short-term scheduling of thermal electricity generating units provide motivation and illustrations. The chapter concludes with directions for future research

Variational Theory for Optimization under Stochastic Ambiguity

This paper is in review.Stochastic ambiguity provides a rich class of uncertainty models that includes those in stochastic, robust, risk-based, and semi-in nite optimization, and that accounts for both uncertainty about parameter values as well as incompleteness of the description of uncertainty. We provide a novel, unifying perspective on optimization under stochastic ambiguity that rests on two pillars. First, the paper models ambiguity by decision-dependent collections of cumulative distribution functions viewed as subsets of a metric space of upper semicontinuous functions. We derive a series of results for this set- ting including estimates of the metric, the hypo-distance, and a new proof of the equivalence with weak convergence. Second, we utilize the theory of lopsided convergence to establish existence, convergence, and approximation of solutions of optimization problems with stochastic ambiguity. For the rst time, we estimate the lop-distance between bifunctions and show that this leads to bounds on the solution quality for problems with stochastic ambiguity. Among other consequences, these results facilitate the study of the \price of robustness" and related quantities

Multivariate Epi-Splines and Evolving Function Identification Problems

Includes erratumThe broad class of extended real-valued lower semicontinuous (lsc) functions on IRn captures nearly all functions of practical importance in equation solving, variational problems, fitting, and estimation. The paper develops piecewise polynomial functions, called epi-splines, that approximate any lsc function to an arbitrary level of accuracy. Epi-splines provide the foundation for the solution of a rich class of function identification problems that incorporate general constraints on the function to be identified including those derived from information about smoothness, shape, proximity to other functions, and so on. As such extrinsic information as well as observed function and subgradient values often evolve in applications, we establish conditions under which the computed epi-splines converge to the function we seek to identify. Numerical examples in response surface building and probability density estimation illustrate the framework.U. S. Army Research Laboratory and the U. S. Army Research Office grant 00101-80683U. S. Army Research Laboratory and the U. S. Army Research Office grant W911NF-10-1-0246U. S. Army Research Laboratory and the U. S. Army Research Office grant W911NF-12-1-0273U. S. Army Research Laboratory and the U. S. Army Research Office grant 00101-80683U. S. Army Research Laboratory and the U. S. Army Research Office grant W911NF-10-1-0246U. S. Army Research Laboratory and the U. S. Army Research Office grant W911NF-12-1-027

Optimality functions and lopsided convergence

The article of record as published may be found at http://dx.doi.org/10.1007/s10957-015-0839-0Optimality functions pioneered by E. Polak characterize stationary points, quantify the degree with which a point fails to be stationary, and play central roles in algorithm development. For optimization problems requiring approximations, optimality functions can be used to ensure consistency in approximations, with the consequence that optimal and stationary points of the approximate problems indeed are approximately optimal and stationary for an original problem. In this paper, we review the framework and illustrate its application to nonlinear programming and other areas. Moreover, we introduce lopsided convergence of bifunctions on metric spaces and show that this notion of convergence is instrumental in establishing consistency of approximations. Lopsided convergence also leads to further characterizations of stationary points under perturbations and approximations.This material is based upon work supported in part by the US Army Research Laboratory and the US Army Research Office under Grant Numbers 00101-80683, W911NF-10-1-0246 and W911NF-12-1-0273.This material is based upon work supported in part by the US Army Research Laboratory and the US Army Research Office under Grant Numbers 00101-80683, W911NF-10-1-0246 and W911NF-12-1-0273