Estimating Mixture Entropy with Pairwise Distances

Mixture distributions arise in many parametric and non-parametric settings -- for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this quantity has no closed-form expression, making some form of approximation necessary. We propose a family of estimators based on a pairwise distance function between mixture components, and show that this estimator class has many attractive properties. For many distributions of interest, the proposed estimators are efficient to compute, differentiable in the mixture parameters, and become exact when the mixture components are clustered. We prove this family includes lower and upper bounds on the mixture entropy. The Chernoff $\alpha$-divergence gives a lower bound when chosen as the distance function, with the Bhattacharyya distance providing the tightest lower bound for components that are symmetric and members of a location family. The Kullback-Leibler divergence gives an upper bound when used as the distance function. We provide closed-form expressions of these bounds for mixtures of Gaussians, and discuss their applications to the estimation of mutual information. We then demonstrate that our bounds are significantly tighter than well-known existing bounds using numeric simulations. This estimator class is very useful in optimization problems involving maximization/minimization of entropy and mutual information, such as MaxEnt and rate distortion problems.Comment: Corrects several errata in published version, in particular in Section V (bounds on mutual information

(p -Cymene)thioglycollatoruthenium(II) dimer; a complex with an ambi-basic S,O-donor ligand

The title compound was prepared from the (p-cymene)ruthenium chloride dimer and thioglycollic acid. The structure is a centrosymmetric dimer bridged by the soft-base S atoms, with the hard-base O atoms of the carboxylate group chelating to form a five-membered twisted-ring. The coordination of the ruthenium atoms is completed by a η6-p-cymene ligand, giving an 18-electron count. The Ru–S bonds are essentially equal at 2.396(1) Å

Nonlinear Information Bottleneck

Information bottleneck (IB) is a technique for extracting information in one random variable $X$ that is relevant for predicting another random variable $Y$. IB works by encoding $X$ in a compressed "bottleneck" random variable $M$ from which $Y$ can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete $X$ and $Y$ with small state spaces, and continuous $X$ and $Y$ with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous $X$ and $Y$, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed "variational IB" method on several real-world datasets

Caveats for information bottleneck in deterministic scenarios

Information bottleneck (IB) is a method for extracting information from one random variable $X$ that is relevant for predicting another random variable $Y$. To do so, IB identifies an intermediate "bottleneck" variable $T$ that has low mutual information $I(X;T)$ and high mutual information $I(Y;T)$. The "IB curve" characterizes the set of bottleneck variables that achieve maximal $I(Y;T)$ for a given $I(X;T)$, and is typically explored by maximizing the "IB Lagrangian", $I(Y;T) - \beta I(X;T)$. In some cases, $Y$ is a deterministic function of $X$, including many classification problems in supervised learning where the output class $Y$ is a deterministic function of the input $X$. We demonstrate three caveats when using IB in any situation where $Y$ is a deterministic function of $X$: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of $\beta$; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. We also show that when $Y$ is a small perturbation away from being a deterministic function of $X$, these three caveats arise in an approximate way. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the three caveats on the MNIST dataset

Pigment analysis by Raman microscopy and portable X-ray fluorescence (pXRF) of thirteenth to fourteenth century illuminations and cuttings from Bologna

Non-destructive pigment analysis by Raman microscopy (RM) and portable X-ray fluorescence (pXRF) has been carried out on some Bolognese illuminations and cuttings chosen to represent the beginnings, evolution and height of Bolognese illuminated manuscript production. Dating to the thirteenth and fourteenth centuries and held in a private collection, the study provides evidence for the pigments generally used in this period. The results, which are compared with those obtained for other north Italian artwork, show the developments in usage of artistic materials and technique. Also addressed in this study is an examination of the respective roles of RM and pXRF analysis in this area of technical art history

Adverse events following influenza immunization reported by healthcare personnel using active surveillance based on text messages

Studies have demonstrated that healthcare personnel (HCP) have concerns about the potential side effects of trivalent inactivate influenza vaccine (IIV3).1-3 A recent metaanalysis of reasons HCP refuse IIV3 indicates the strongest predictors of vaccine acceptance are belief that the vaccine is safe and belief the vaccine does not cause the disease it is meant to prevent.

Use of an index to reflect the aggregate burden of long-term exposure to criteria air pollutants in the United States.

Air pollution control in the United States for five common pollutants--particulate matter, ground-level ozone, sulfur dioxide, nitrogen dioxide, and carbon monoxide--is based partly on the attainment of ambient air quality standards that represent a level of air pollution regarded as safe. Regulatory and health agencies often focus on whether standards for short periods are attained; the number of days that standards are exceeded is used to track progress. Efforts to explain air pollution to the public often incorporate an air quality index that represents daily concentrations of pollutants. While effects of short-term exposures have been emphasized, research shows that long-term exposures to lower concentrations of air pollutants can also result in adverse health effects. We developed an aggregate index that represents long-term exposure to these pollutants, using 1995 monitoring data for metropolitan areas obtained from the U.S. Environmental Protection Agency's Aerometric Information Retrieval System. We compared the ranking of metropolitan areas under the proposed aggregate index with the ranking of areas by the number of days that short-term standards were exceeded. The geographic areas with the highest burden of long-term exposures are not, in all cases, the same as those with the most days that exceeded a short-term standard. We believe that an aggregate index of long-term air pollution offers an informative addition to the principal approaches currently used to describe air pollution exposures; further work on an aggregate index representing long-term exposure to air pollutants is warranted