Segmentation Propagation in ImageNet

Abstract. ImageNet is a large-scale hierarchical database of object classes. We propose to automatically populate it with pixelwise segmentations, by leveraging existing manual annotations in the form of class labels and bounding-boxes. The key idea is to recursively exploit images segmented so far to guide the segmentation of new images. At each stage this propagation process expands into the images which are easiest to segment at that point in time, e.g. by moving to the semantically most related classes to those segmented so far. The propagation of segmentation occurs both (a) at the image level, by transferring existing segmentations to estimate the probability of a pixel to be foreground, and (b) at the class level, by jointly segmenting images of the same class and by importing the appearance models of classes that are already segmented. Through an experiment on 577 classes and 500k images we show that our technique (i) annotates a wide range of classes with accurate segmentations; (ii) effectively exploits the hierarchical structure of ImageNet; (iii) scales efficiently; (iv) outperforms a baseline GrabCut [1] initialized on the image center, as well as our recent segmentation transfer technique [2] on which this paper is based. Moreover, our method also delivers state-of-the-art results on the recent iCoseg dataset for co-segmentation.

Real-Time Face Pose Estimation from Single Range Images

We present a real-time algorithm to estimate the 3D pose of a previously unseen face from a single range image. Based on a novel shape signature to identify noses in range images, we generate candidates for their positions, and then generate and evaluate many pose hypotheses in parallel using modern graphics processing units (GPUs). We developed a novel error function that compares the input range image to precomputed pose images of an average face model. The algorithm is robust to large pose variations of plusmn90deg yaw, plusmn45deg pitch and plusmn30deg roll rotation, facial expression, partial occlusion, and works for multiple faces in the field of view. It correctly estimates 97.8% of the poses within yaw and pitch error of 15deg at 55.8 fps. To evaluate the algorithm, we built a database of range images with large pose variations and developed a method for automatic ground truth annotation.Engineering and Applied Science

Joint Inference in Weakly-Annotated Image Datasets via Dense Correspondence

We present a principled framework for inferring pixel labels in weakly-annotated image datasets. Most previous, example-based approaches to computer vision rely on a large corpus of densely labeled images. However, for large, modern image datasets, such labels are expensive to obtain and are often unavailable. We establish a large-scale graphical model spanning all labeled and unlabeled images, then solve it to infer pixel labels jointly for all images in the dataset while enforcing consistent annotations over similar visual patterns. This model requires significantly less labeled data and assists in resolving ambiguities by propagating inferred annotations from images with stronger local visual evidences to images with weaker local evidences. We apply our proposed framework to two computer vision problems, namely image annotation with semantic segmentation, and object discovery and co-segmentation (segmenting multiple images containing a common object). Extensive numerical evaluations and comparisons show that our method consistently outperforms the state-of-the-art in automatic annotation and semantic labeling, while requiring significantly less labeled data. In contrast to previous co-segmentation techniques, our method manages to discover and segment objects well even in the presence of substantial amounts of noise images (images not containing the common object), as typical for datasets collected from Internet search

Bridge to the stars: A mission concept to an interstellar object

Exoplanet discoveries since the mid-1990’s have revealed an astounding diversity of planetary systems. Studying these systems is essential to understanding planetary formation processes, as well as the development of life in the universe. Unfortunately, humanity can only observe limited aspects of exoplanetary systems by telescope, and the significant distances between stars presents a barrier to in situ exploration. In this study, we propose an alternative path to gain insight into exoplanetary systems: Bridge, a mission concept design to fly by an interstellar object as it passes through our solar system. Designed as a New Frontiers-class mission during the National Aeronautics and Space Administration (NASA) Planetary Science Summer School, Bridge would provide a unique opportunity to gain insight into potential physical, chemical, and biological differences between solar systems as well as the possible exchange of planetary materials between them. Bridge employs ultraviolet/visible, near-infrared, and mid-infrared point spectrometers, a visible camera, and a guided impactor. We also provide a quantitative Monte Carlo analysis that estimates wait times for a suitable target, and examines key trades between ground storage and a parking orbit, power sources, inner versus outer solar system encounters, and launch criteria. Due to the fleeting nature of interstellar objects, reaching an interstellar object may require an extended ground storage phase for the spacecraft until a suitable ISO is discovered, followed by a rapid response launch strategy. To enable rapid response missions designed to intercept such unique targets, language would need to be added to future NASA announcements of opportunity such that ground storage and rapid response would be allowable components of a proposed mission

Learning to approximate global shape priors for figure-ground segmentation

We present a technique for approximate minimization of two-label energy functions with higher-order or global potentials. Our method treats the energy function as a blackbox: it does not exploit knowledge of its form nor its order, as opposed to optimization schemes specialized to a particular form. The key idea is to automatically learn a lower-order approximation of the energy function, which can then be minimized used existing efficient algorithms. We experimentally demonstrate our method for binary image segmentation, where it enables to incorporate a global shape prior into traditional models based on pairwise conditional random fields