Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

Studying how to fine-tune offline reinforcement learning (RL) pre-trained policy is profoundly significant for enhancing the sample efficiency of RL algorithms. However, directly fine-tuning pre-trained policies often results in sub-optimal performance. This is primarily due to the distribution shift between offline pre-training and online fine-tuning stages. Specifically, the distribution shift limits the acquisition of effective online samples, ultimately impacting the online fine-tuning performance. In order to narrow down the distribution shift between offline and online stages, we proposed Q conditioned state entropy (QCSE) as intrinsic reward. Specifically, QCSE maximizes the state entropy of all samples individually, considering their respective Q values. This approach encourages exploration of low-frequency samples while penalizing high-frequency ones, and implicitly achieves State Marginal Matching (SMM), thereby ensuring optimal performance, solving the asymptotic sub-optimality of constraint-based approaches. Additionally, QCSE can seamlessly integrate into various RL algorithms, enhancing online fine-tuning performance. To validate our claim, we conduct extensive experiments, and observe significant improvements with QCSE (about 13% for CQL and 8% for Cal-QL). Furthermore, we extended experimental tests to other algorithms, affirming the generality of QCSE

Research on Feature Extraction Data Processing System For MRI of Brain Diseases Based on Computer Deep Learning

Most of the existing wavelet image processing techniques are carried out in the form of single-scale reconstruction and multiple iterations. However, processing high-quality fMRI data presents problems such as mixed noise and excessive computation time. This project proposes the use of matrix operations by combining mixed noise elimination methods with wavelet analysis to replace traditional iterative algorithms. Functional magnetic resonance imaging (fMRI) of the auditory cortex of a single subject is analyzed and compared to the wavelet domain signal processing technology based on repeated times and the world's most influential SPM8. Experiments show that this algorithm is the fastest in computing time, and its detection effect is comparable to the traditional iterative algorithm. However, this has a higher practical value for the processing of FMRI data. In addition, the wavelet analysis method proposed signal processing to speed up the calculation rate

Timber production assessment of a plantation forest: An integrated framework with field-based inventory, multi-source remote sensing data and forest management history

Timber production is the purpose for managing plantation forests, and its spatial and quantitative information is critical for advising management strategies. Previous studies have focused on growing stock volume (GSV), which represents the current potential of timber production, yet few studies have investigated historical process-harvested timber. This resulted in a gap in a synthetical ecosystem service assessment of timber production. In this paper, we established a Management Process-based Timber production (MPT) framework to integrate the current GSV and the harvested timber derived from historical logging regimes, trying to synthetically assess timber production for a historical period. In the MPT framework, age-class and current GSV determine the times of historical thinning and the corresponding harvested timber, by using a "space-for-time" substitution. The total timber production can be estimated by the historical harvested timber in each thinning and the current GSV. To test this MPT framework, an empirical study on a larch plantation (LP) with area of 43,946 ha was conducted in North China for a period from 1962 to 2010. Field-based inventory data was integrated with ALOS PALSAR (Advanced Land-Observing Satellite Phased Array L-band Synthetic Aperture Radar) and Landsat-8 OLI (Operational Land Imager) data for estimating the age-class and current GSV of LP. The random forest model with PALSAR backscatter intensity channels and OLI bands as input predictive variables yielded an accuracy of 67.9% with a Kappa coefficient of 0.59 for age-class classification. The regression model using PALSAR data produced a root mean square error (RMSE) of 36.5 m(3) ha(-1). The total timber production of LP was estimated to be 7.27 x 10(6) m(3), with 4.87 x 10(6) m(3) in current GSV and 2.40 x 10(6) m(3) in harvested timber through historical thinning. The historical process-harvested timber accounts to 33.0% of the total timber production, which component has been neglected in the assessments for current status of plantation forests. Synthetically considering the RMSE for predictive GSV and misclassification of age-class, the error in timber production were supposed to range from -55.2 to 56.3 m(3) ha(-1). The MPT framework can be used to assess timber production of other tree species at a larger spatial scale, providing crucial information for a better understanding of forest ecosystem service. (C) 2016 Elsevier B.V. All rights reserved.ArticleINTERNATIONAL JOURNAL OF APPLIED EARTH OBSERVATION AND GEOINFORMATION.52:155-165(2016)journal articl

Are All Losses Created Equal: A Neural Collapse Perspective

While cross entropy (CE) is the most commonly used loss to train deep neural networks for classification tasks, many alternative losses have been developed to obtain better empirical performance. Among them, which one is the best to use is still a mystery, because there seem to be multiple factors affecting the answer, such as properties of the dataset, the choice of network architecture, and so on. This paper studies the choice of loss function by examining the last-layer features of deep networks, drawing inspiration from a recent line work showing that the global optimal solution of CE and mean-square-error (MSE) losses exhibits a Neural Collapse phenomenon. That is, for sufficiently large networks trained until convergence, (i) all features of the same class collapse to the corresponding class mean and (ii) the means associated with different classes are in a configuration where their pairwise distances are all equal and maximized. We extend such results and show through global solution and landscape analyses that a broad family of loss functions including commonly used label smoothing (LS) and focal loss (FL) exhibits Neural Collapse. Hence, all relevant losses(i.e., CE, LS, FL, MSE) produce equivalent features on training data. Based on the unconstrained feature model assumption, we provide either the global landscape analysis for LS loss or the local landscape analysis for FL loss and show that the (only!) global minimizers are neural collapse solutions, while all other critical points are strict saddles whose Hessian exhibit negative curvature directions either in the global scope for LS loss or in the local scope for FL loss near the optimal solution. The experiments further show that Neural Collapse features obtained from all relevant losses lead to largely identical performance on test data as well, provided that the network is sufficiently large and trained until convergence.Comment: 32 page, 10 figures, NeurIPS 202

Developing an explainable deep learning module based on the LSTM framework for flood prediction

Long short-term memory (LSTM) networks have become indispensable tools in hydrological modeling due to their ability to capture long-term dependencies, handle non-linear relationships, and integrate multiple data sources but suffer from limited interpretability due to their black box nature. To address this limitation, we propose an explainable module within the LSTM framework, specifically designed for flood prediction across 531 catchments in the contiguous United States. Our approach incorporates a simplified gated module, which is interposed between the input data and the LSTM network, providing a transparent view of the module’s pattern recognition process. This gated module allows for easy identification of key variables and optimal lookback windows, and clusters the gated information into four categories: short-term and long-term impacts of precipitation and temperature. This categorization enhances our understanding of how the module utilizes input data and reveals underlying mechanisms in flood prediction. The modular design of our approach demonstrates high correlation with Saliency method, validating the credibility of its explanatory mechanisms, providing comparable interpretability features to LSTMs while illuminating key variables and optimal lookback windows considered most informative by hydrological models, and opening up new avenues for AI-assisted scientific discovery in the field

Regulation of Two-Dimensional Lattice Deformation Recovery

The lattice directly determines the electronic structure, and it enables controllably tailoring the properties by deforming the lattices of two-dimensional (2D)materials. Owing to the unbalanced electrostatic equilibrium among the dislocated atoms, the deformed lattice is thermodynamically unstable and would recover to the initial state. Here, we demonstrate that the recovery of deformed 2D lattices could be directly regulated via doping metal donors to reconstruct electrostatic equilibrium. Compared with the methods that employed external force fields with intrinsic instability and nonuniformity, the stretched 2D molybdenum diselenide (MoSe2)could be uniformly retained and permanently preserved via doping metal atoms with more outermost electrons and smaller electronegativity than Mo. We believe that the proposed strategy could open up a new avenue in directly regulating the atomic-thickness lattice and promote its practical applications based on 2D crystals. © 2019 The Author(s

Influence of Chrysanthemum morifolium-maize intercropping pattern on yield, quality, soil condition, and rhizosphere soil microbial communities of C. morifolium

IntroductionChrysanthemum morifolium Ramat. is a perennial herb in the Compositae family, often employed in traditional Chinese medicine due to its medicinal value. The planting of C. morifolium faces the challenges of continuous cropping, and intercropping is able to somewhat overcome the obstacles of continuous cropping.MethodsIn our study, we designed two different C. morifolium-maize intercropping patterns, including C. morifolium-maize narrow-wide row planting (IS) and C. morifolium-maize middle row planting (IM). Compared with monoculture, the agronomic traits, yield, active ingredients, soil physicochemical properties, soil enzyme activities, and rhizosphere soil microbial communities of C. morifolium and maize were measured under the two C. morifolium-maize intercropping patterns.ResultsThe findings indicated that (1) Intercropping elevated the agronomic traits, yield, and active ingredients of C. morifolium, especially in C. morifolium-maize narrow-wide row planting pattern, which indicating that interspecific distance played an important role in intercropping system; (2) Intercropping enhanced soil physicochemical properties and enzyme activities of C. morifolium and maize; (3) Intercropping altered rhizosphere soil microbial communities of C. morifolium and maize, making microbial interrelationships more complex. (4) Intercropping could recruit a large number of beneficial microorganisms enrich in the soil, including Bacillus, Sphingomonas, Burkholderia-Caballeronia-Paraburkholderia, Chaetomium, and Ceratorhiza, which may increase the content of AN, NN, AvK, ExCa, AvCu, AvZn and other nutrients in soil and promoted the growth and quality of C. morifolium.DiscussionIn summary, intercropping with maize could promote the accumulation of beneficial microorganisms in the soil, thus improving the overall growing environment, and finally realizing the growth and improvement of C. morifolium