CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition

Most of the traditional work on intrinsic image decomposition rely on deriving priors about scene characteristics. On the other hand, recent research use deep learning models as in-and-out black box and do not consider the well-established, traditional image formation process as the basis of their intrinsic learning process. As a consequence, although current deep learning approaches show superior performance when considering quantitative benchmark results, traditional approaches are still dominant in achieving high qualitative results. In this paper, the aim is to exploit the best of the two worlds. A method is proposed that (1) is empowered by deep learning capabilities, (2) considers a physics-based reflection model to steer the learning process, and (3) exploits the traditional approach to obtain intrinsic images by exploiting reflectance and shading gradient information. The proposed model is fast to compute and allows for the integration of all intrinsic components. To train the new model, an object centered large-scale datasets with intrinsic ground-truth images are created. The evaluation results demonstrate that the new model outperforms existing methods. Visual inspection shows that the image formation loss function augments color reproduction and the use of gradient information produces sharper edges. Datasets, models and higher resolution images are available at https://ivi.fnwi.uva.nl/cv/retinet.Comment: CVPR 201

Identifiability of dynamical networks: which nodes need be measured?

Much recent research has dealt with the identifiability of a dynamical network in which the node signals are connected by causal linear time-invariant transfer functions and are possibly excited by known external excitation signals and/or unknown noise signals. So far all results on the identifiability of the whole network have assumed that all node signals are measured. Under this assumption, it has been shown that such networks are identifiable only if some prior knowledge is available about the structure of the network, in particular the structure of the excitation. In this paper we present the first results for the situation where not all node signals are measurable, under the assumptions that the topology of the network is known, that each node is excited by a known signal and that the nodes are noise-free. Using graph theoretical properties, we show that the transfer functions that can be identified depend essentially on the topology of the paths linking the corresponding vertices to the measured nodes. An important outcome of our research is that, under those assumptions, a network can often be identified using only a small subset of node measurements.Comment: extended version of an article to appear at the 56th IEEE Conference on Decision and Control, CDC 2017 8 pages, 2 style files, 3 pdf figures, 2 png figure

Relating the metatranscriptome and metagenome of the human gut

Although the composition of the human microbiome is now wellstudied, the microbiota’s \u3e8 million genes and their regulation remain largely uncharacterized. This knowledge gap is in part because of the difficulty of acquiring large numbers of samples amenable to functional studies of the microbiota. We conducted what is, to our knowledge, one of the first human microbiome studies in a well-phenotyped prospective cohort incorporating taxonomic, metagenomic, and metatranscriptomic profiling at multiple body sites using self-collected samples. Stool and saliva were provided by eight healthy subjects, with the former preserved by three different methods (freezing, ethanol, and RNAlater) to validate self-collection. Within-subject microbial species, gene, and transcript abundances were highly concordant across sampling methods, with only a small fraction of transcripts (\u3c5%) displaying between-method variation. Next, we investigated relationships between the oral and gut microbial communities, identifying a subset of abundant oral microbes that routinely survive transit to the gut, but with minimal transcriptional activity there. Finally, systematic comparison of the gut metagenome and metatranscriptome revealed that a substantial fraction (41%) of microbial transcripts were not differentially regulated relative to their genomic abundances. Of the remainder, consistently underexpressed pathways included sporulation and amino acid biosynthesis, whereas up-regulated pathways included ribosome biogenesis and methanogenesis. Across subjects, metatranscriptional profiles were significantly more individualized than DNA-level functional profiles, but less variable than microbial composition, indicative of subject-specific whole-community regulation. The results thus detail relationships between community genomic potential and gene expression in the gut, and establish the feasibility of metatranscriptomic investigations in subject-collected and shipped samples

Metagenomic biomarker discovery and explanation

This study describes and validates a new method for metagenomic biomarker discovery by way of class comparison, tests of biological consistency and effect size estimation. This addresses the challenge of finding organisms, genes, or pathways that consistently explain the differences between two or more microbial communities, which is a central problem to the study of metagenomics. We extensively validate our method on several microbiomes and a convenient online interface for the method is provided at http://huttenhower.sph.harvard.edu/lefse/.National Institute of Dental and Craniofacial Research (U.S.) (grant DE017106)National Institutes of Health (U.S.) (NIH grant AI078942)Burroughs Wellcome FundNational Institutes of Health (U.S.) (NIH 1R01HG005969

Sub-clinical detection of gut microbial biomarkers of obesity and type 2 diabetes

Background: Obesity and type 2 diabetes (T2D) are linked both with host genetics and with environmental factors, including dysbioses of the gut microbiota. However, it is unclear whether these microbial changes precede disease onset. Twin cohorts present a unique genetically-controlled opportunity to study the relationships between lifestyle factors and the microbiome. In particular, we hypothesized that family-independent changes in microbial composition and metabolic function during the sub-clinical state of T2D could be either causal or early biomarkers of progression. Methods: We collected fecal samples and clinical metadata from 20 monozygotic Korean twins at up to two time points, resulting in 36 stool shotgun metagenomes. While the participants were neither obese nor diabetic, they spanned the entire range of healthy to near-clinical values and thus enabled the study of microbial associations during sub-clinical disease while accounting for genetic background. Results: We found changes both in composition and in function of the sub-clinical gut microbiome, including a decrease in Akkermansia muciniphila suggesting a role prior to the onset of disease, and functional changes reflecting a response to oxidative stress comparable to that previously observed in chronic T2D and inflammatory bowel diseases. Finally, our unique study design allowed us to examine the strain similarity between twins, and we found that twins demonstrate strain-level differences in composition despite species-level similarities. Conclusions: These changes in the microbiome might be used for the early diagnosis of an inflamed gut and T2D prior to clinical onset of the disease and will help to advance toward microbial interventions. Electronic supplementary material The online version of this article (doi:10.1186/s13073-016-0271-6) contains supplementary material, which is available to authorized users

Intrinsic image decomposition using physics-based cues and CNNs

Intrinsic image decomposition is the decomposition of an image into its reflectance and shading components. The intrinsic image decomposition problem is inherently ill-posed, since there can be multiple solutions to compute the intrinsic components forming the same image. In this paper, we explore the use of physics-based priors. We also propose a new architecture that separates the learning components in a stacked manner. We explore various ways of integrating such priors into a deep learning system. Our method is trained and tested on a large synthetic garden dataset to assess its performance. It is evaluated and compared to state-of-the-art methods using two standard intrinsic datasets. Finally, the pre-trained network is tested on real world images to show the generalisation capabilities of the network

Learning Generalized Segmentation for Foggy-scenes by Bi-directional Wavelet Guidance

Learning scene semantics that can be well generalized to foggy conditions is important for safety-crucial applications such as autonomous driving. Existing methods need both annotated clear images and foggy images to train a curriculum domain adaptation model. Unfortunately, these methods can only generalize to the target foggy domain that has seen in the training stage, but the foggy domains vary a lot in both urban-scene styles and fog styles. In this paper, we propose to learn scene segmentation well generalized to foggy-scenes under the domain generalization setting, which does not involve any foggy images in the training stage and can generalize to any arbitrary unseen foggy scenes. We argue that an ideal segmentation model that can be well generalized to foggy-scenes need to simultaneously enhance the content, de-correlate the urban-scene style and de-correlate the fog style. As the content (e.g., scene semantic) rests more in low-frequency features while the style of urban-scene and fog rests more in high-frequency features, we propose a novel bi-directional wavelet guidance (BWG) mechanism to realize the above three objectives in a divide-and-conquer manner. With the aid of Haar wavelet transformation, the low frequency component is concentrated on the content enhancement self-attention, while the high frequency component is shifted to the style and fog self-attention for de-correlation purpose. It is integrated into existing mask-level Transformer segmentation pipelines in a learnable fashion. Large-scale experiments are conducted on four foggy-scene segmentation datasets under a variety of interesting settings. The proposed method significantly outperforms existing directly-supervised, curriculum domain adaptation and domain generalization segmentation methods. Source code is available at https://github.com/BiQiWHU/BWG

Learning Content-Enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation

Domain-generalized urban-scene semantic segmentation (USSS) aims to learn generalized semantic predictions across diverse urban-scene styles. Unlike generic domain gap challenges, USSS is unique in that the semantic categories are often similar in different urban scenes, while the styles can vary significantly due to changes in urban landscapes, weather conditions, lighting, and other factors. Existing approaches typically rely on convolutional neural networks (CNNs) to learn the content of urban scenes.In this paper, we propose a Content-enhanced Mask TransFormer (CMFormer) for domain-generalized USSS. The main idea is to enhance the focus of the fundamental component, the mask attention mechanism, in Transformer segmentation models on content information. We have observed through empirical analysis that a mask representation effectively captures pixel segments, albeit with reduced robustness to style variations. Conversely, its lower-resolution counterpart exhibits greater ability to accommodate style variations, while being less proficient in representing pixel segments. To harness the synergistic attributes of these two approaches, we introduce a novel content-enhanced mask attention mechanism. It learns mask queries from both the image feature and its down-sampled counterpart, aiming to simultaneously encapsulate the content and address stylistic variations. These features are fused into a Transformer decoder and integrated into a multi-resolution content-enhanced mask attention learning scheme.Extensive experiments conducted on various domain-generalized urban-scene segmentation datasets demonstrate that the proposed CMFormer significantly outperforms existing CNN-based methods by up to 14.0% mIoU and the contemporary HGFormer by up to 1.7% mIoU. The source code is publicly available at https://github.com/BiQiWHU/CMFormer