Learnable Reconstruction Methods from RGB Images to Hyperspectral Imaging: A Survey

Hyperspectral imaging enables versatile applications due to its competence in capturing abundant spatial and spectral information, which are crucial for identifying substances. However, the devices for acquiring hyperspectral images are expensive and complicated. Therefore, many alternative spectral imaging methods have been proposed by directly reconstructing the hyperspectral information from lower-cost, more available RGB images. We present a thorough investigation of these state-of-the-art spectral reconstruction methods from the widespread RGB images. A systematic study and comparison of more than 25 methods has revealed that most of the data-driven deep learning methods are superior to prior-based methods in terms of reconstruction accuracy and quality despite lower speeds. This comprehensive review can serve as a fruitful reference source for peer researchers, thus further inspiring future development directions in related domains

Freeform optical system design with differentiable three-dimensional ray tracing and unsupervised learning

Optical systems have been crucial for versatile applications such as consumer electronics, remote sensing and biomedical imaging. Designing optical systems has been a highly professional work due to complicated aberration theories and intangible rules-of-thumb, hence neural networks are only coming into this realm until recent years. In this work, we propose and implement a generic, differentiable freeform raytracing module, suitable for off-axis, multiple-surface freeform/aspheric optical systems, paving the way toward a deep learning-based optical design method. The network is trained with minimal prior knowledge, and it can infer numerous optical systems after a one-time training. The presented work unlocks great potential for deep learning in various freeform/aspheric optical systems, and the trained network could serve as an effective, unified platform for generating, recording, and replicating good initial optical designs.</jats:p

A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging

AbstractHyperspectral imaging enables many versatile applications for its competence in capturing abundant spatial and spectral information, which is crucial for identifying substances. However, the devices for acquiring hyperspectral images are typically expensive and very complicated, hindering the promotion of their application in consumer electronics, such as daily food inspection and point-of-care medical screening, etc. Recently, many computational spectral imaging methods have been proposed by directly reconstructing the hyperspectral information from widely available RGB images. These reconstruction methods can exclude the usage of burdensome spectral camera hardware while keeping a high spectral resolution and imaging performance. We present a thorough investigation of more than 25 state-of-the-art spectral reconstruction methods which are categorized as prior-based and data-driven methods. Simulations on open-source datasets show that prior-based methods are more suitable for rare data situations, while data-driven methods can unleash the full potential of deep learning in big data cases. We have identified current challenges faced by those methods (e.g., loss function, spectral accuracy, data generalization) and summarized a few trends for future work. With the rapid expansion in datasets and the advent of more advanced neural networks, learnable methods with fine feature representation abilities are very promising. This comprehensive review can serve as a fruitful reference source for peer researchers, thus paving the way for the development of computational hyperspectral imaging.</jats:p

Supplementary document for Freeform optical system design with differentiable three-dimensional ray tracing and unsupervised learning - 6280891.pdf

Supplementary information for raytracing and overlap detectio