Non-ergodicity in reinforcement learning: robustness via ergodicity transformations

Envisioned application areas for reinforcement learning (RL) include autonomous driving, precision agriculture, and finance, which all require RL agents to make decisions in the real world. A significant challenge hindering the adoption of RL methods in these domains is the non-robustness of conventional algorithms. In this paper, we argue that a fundamental issue contributing to this lack of robustness lies in the focus on the expected value of the return as the sole ``correct'' optimization objective. The expected value is the average over the statistical ensemble of infinitely many trajectories. For non-ergodic returns, this average differs from the average over a single but infinitely long trajectory. Consequently, optimizing the expected value can lead to policies that yield exceptionally high returns with probability zero but almost surely result in catastrophic outcomes. This problem can be circumvented by transforming the time series of collected returns into one with ergodic increments. This transformation enables learning robust policies by optimizing the long-term return for individual agents rather than the average across infinitely many trajectories. We propose an algorithm for learning ergodicity transformations from data and demonstrate its effectiveness in an instructive, non-ergodic environment and on standard RL benchmarks

Silencing microRNA-134 produces neuroprotective and prolonged seizure-suppressive effects

Temporal lobe epilepsy is a common, chronic neurological disorder characterized by recurrent spontaneous seizures. MicroRNAs (miRNAs) are small, noncoding RNAs that regulate post-transcriptional expression of protein-coding mRNAs, which may have key roles in the pathogenesis of neurological disorders. In experimental models of prolonged, injurious seizures (status epilepticus) and in human epilepsy, we found upregulation of miR-134, a brain-specific, activity-regulated miRNA that has been implicated in the control of dendritic spine morphology. Silencing of miR-134 expression in vivo using antagomirs reduced hippocampal CA3 pyramidal neuron dendrite spine density by 21% and rendered mice refractory to seizures and hippocampal injury caused by status epilepticus. Depletion of miR-134 after status epilepticus in mice reduced the later occurrence of spontaneous seizures by over 90% and mitigated the attendant pathological features of temporal lobe epilepsy. Thus, silencing miR-134 exerts prolonged seizure-suppressant and neuroprotective actions; determining whether these are anticonvulsant effects or are truly antiepileptogenic effects requires additional experimentation

CTDAS-Lagrange v1.0:a high-resolution data assimilation system for regional carbon dioxide observations

We have implemented a regional carbon dioxide data assimilation system based on the CarbonTracker Data Assimilation Shell (CTDAS) and a high-resolution Lagrangian transport model, the Stochastic Time-Inverted Lagrangian Transport model driven by the Weather Forecast and Research meteorological fields (WRF-STILT). With this system, named CTDAS-Lagrange, we simultaneously optimize terrestrial biosphere fluxes and four parameters that adjust the lateral boundary conditions (BCs) against CO2 observations from the NOAA ESRL North America tall tower and aircraft programmable flask packages (PFPs) sampling program. Least-squares optimization is performed with a time-stepping ensemble Kalman smoother, over a time window of 10 days and assimilating sequentially a time series of observations. Because the WRF-STILT footprints are pre-computed, it is computationally efficient to run the CTDAS-Lagrange system.To estimate the uncertainties in the optimized fluxes from the system, we performed sensitivity tests with various a priori biosphere fluxes (SiBCASA, SiB3, CT2013B) and BCs (optimized mole fraction fields from CT2013B and CTE2014, and an empirical dataset derived from aircraft observations), as well as with a variety of choices on the ways that fluxes are adjusted (additive or multiplicative), covariance length scales, biosphere flux covariances, BC parameter uncertainties, and model-data mismatches. In pseudo-data experiments, we show that in our implementation the additive flux adjustment method is more flexible in optimizing net ecosystem exchange (NEE) than the multiplicative flux adjustment method, and our sensitivity tests with real observations show that the CTDAS-Lagrange system has the ability to correct for the potential biases in the lateral BCs and to resolve large biases in the prior biosphere fluxes.Using real observations, we have derived a range of estimates for the optimized carbon fluxes from a series of sensitivity tests, which places the North American carbon sink for the year 2010 in a range from -0.92 to -1.26 PgC yr( -1). This is comparable to the TM5-based estimates of CarbonTracker (version CT2016, -0.91 +/- 1.10 PgC yr (-1)) and CarbonTracker Europe (version CTE,2016, -0.91 +/- 0.31 PgC yr(-1)). We conclude that CTDAS-Lagrange can offer a versatile and computationally attractive alternative to these global systems for regional estimates of carbon fluxes, which can take advantage of high-resolution Lagrangian footprints that are increasingly easy to obtain

Reduced levels of two modifiers of epigenetic gene silencing, Dnmt3a and Trim28, cause increased phenotypic noise

Background: Inbred individuals reared in controlled environments display considerable variance in many complex traits but the underlying cause of this intangible variation has been an enigma. Here we show that two modifiers of epigenetic gene silencing play a critical role in the process.Results: Inbred mice heterozygous for a null mutation in DNA methyltransferase 3a (Dnmt3a) or tripartite motif protein 28 (Trim28) show greater coefficients of variance in body weight than their wild-type littermates. Trim28 mutants additionally develop metabolic syndrome and abnormal behavior with incomplete penetrance. Genome-wide gene expression analyses identified 284 significantly dysregulated genes in Trim28 heterozygote mutants compared to wild-type mice, with Mas1, which encodes a G-protein coupled receptor implicated in lipid metabolism, showing the greatest average change in expression (7.8-fold higher in mutants). This gene also showed highly variable expression between mutant individuals.Conclusions: These studies provide a molecular explanation of developmental noise in whole organisms and suggest that faithful epigenetic control of transcription is central to suppressing deleterious levels of phenotypic variation. These findings have broad implications for understanding the mechanisms underlying sporadic and complex disease in humans

CTDAS-Lagrange v1.0 : A high-resolution data assimilation system for regional carbon dioxide observations

We have implemented a regional carbon dioxide data assimilation system based on the CarbonTracker Data Assimilation Shell (CTDAS) and a high-resolution Lagrangian transport model, the Stochastic Time-Inverted Lagrangian Transport model driven by the Weather Forecast and Research meteorological fields (WRF-STILT). With this system, named CTDAS-Lagrange, we simultaneously optimize terrestrial biosphere fluxes and four parameters that adjust the lateral boundary conditions (BCs) against CO2 observations from the NOAA ESRL North America tall tower and aircraft programmable flask packages (PFPs) sampling program. Least-squares optimization is performed with a time-stepping ensemble Kalman smoother, over a time window of 10 days and assimilating sequentially a time series of observations. Because the WRF-STILT footprints are pre-computed, it is computationally efficient to run the CTDAS-Lagrange system. To estimate the uncertainties in the optimized fluxes from the system, we performed sensitivity tests with various a priori biosphere fluxes (SiBCASA, SiB3, CT2013B) and BCs (optimized mole fraction fields from CT2013B and CTE2014, and an empirical dataset derived from aircraft observations), as well as with a variety of choices on the ways that fluxes are adjusted (additive or multiplicative), covariance length scales, biosphere flux covariances, BC parameter uncertainties, and model-data mismatches. In pseudo-data experiments, we show that in our implementation the additive flux adjustment method is more flexible in optimizing net ecosystem exchange (NEE) than the multiplicative flux adjustment method, and our sensitivity tests with real observations show that the CTDAS-Lagrange system has the ability to correct for the potential biases in the lateral BCs and to resolve large biases in the prior biosphere fluxes. Using real observations, we have derived a range of estimates for the optimized carbon fluxes from a series of sensitivity tests, which places the North American carbon sink for the year 2010 in a range from -0.92 to -1.26PgCyr-1. This is comparable to the TM5-based estimates of CarbonTracker (version CT2016, -0.91±1.10PgCyr-1) and CarbonTracker Europe (version CTE2016, -0.91±0.31PgCyr-1). We conclude that CTDAS-Lagrange can offer a versatile and computationally attractive alternative to these global systems for regional estimates of carbon fluxes, which can take advantage of high-resolution Lagrangian footprints that are increasingly easy to obtain.</p

Global Carbon Budget 2022

Accurate assessment of anthropogenic carbon dioxide (CO$_2$) emissions and their redistribution among the atmosphere, ocean, and terrestrial biosphere in a changing climate is critical to better understand the global carbon cycle, support the development of climate policies, and project future climate change. Here we describe and synthesize data sets and methodologies to quantify the five major components of the global carbon budget and their uncertainties. Fossil CO$_2$ emissions (E$_{FOS}$) are based on energy statistics and cement production data, while emissions from land-use change (E$_{LUC}$), mainly deforestation, are based on land use and land-use change data and bookkeeping models. Atmospheric CO$_2$ concentration is measured directly, and its growth rate (G$_{ATM}$) is computed from the annual changes in concentration. The ocean CO$_2$ sink (S$_{OCEAN}$) is estimated with global ocean biogeochemistry models and observation-based data products. The terrestrial CO$_2$ sink (S$_{LAND}$) is estimated with dynamic global vegetation models. The resulting carbon budget imbalance (B$_{IM}$), the difference between the estimated total emissions and the estimated changes in the atmosphere, ocean, and terrestrial biosphere, is a measure of imperfect data and understanding of the contemporary carbon cycle. All uncertainties are reported as ±1σ. For the year 2021, E$_{FOS}$ increased by 5.1 % relative to 2020, with fossil emissions at 10.1 ± 0.5 GtC yr$^{−1}$ (9.9 ± 0.5 GtC yr$^{−1}$ when the cement carbonation sink is included), and E$_{LUC}$ was 1.1 ± 0.7 GtC yr$^{−1}$, for a total anthropogenic CO$_2$ emission (including the cement carbonation sink) of 10.9 ± 0.8 GtC yr$^{−1}$ (40.0 ± 2.9 GtCO$_2$). Also, for 2021, G$_{ATM}$ was 5.2 ± 0.2 GtC yr$^{−1}$ (2.5 ± 0.1 ppm yr$^{−1}$), S$_{OCEAN}$ was 2.9  ± 0.4 GtC yr$^{−1}$, and S$_{LAND}$ was 3.5 ± 0.9 GtC yr$^{−1}$, with a B$_{IM}$ of −0.6 GtC yr$^{−1}$ (i.e. the total estimated sources were too low or sinks were too high). The global atmospheric CO$_2$ concentration averaged over 2021 reached 414.71 ± 0.1 ppm. Preliminary data for 2022 suggest an increase in E$_{FOS}$ relative to 2021 of +1.0 % (0.1 % to 1.9 %) globally and atmospheric CO$_2$ concentration reaching 417.2 ppm, more than 50 % above pre-industrial levels (around 278 ppm). Overall, the mean and trend in the components of the global carbon budget are consistently estimated over the period 1959–2021, but discrepancies of up to 1 GtC yr$^{−1}$ persist for the representation of annual to semi-decadal variability in CO$_2$ fluxes. Comparison of estimates from multiple approaches and observations shows (1) a persistent large uncertainty in the estimate of land-use change emissions, (2) a low agreement between the different methods on the magnitude of the land CO$_2$ flux in the northern extratropics, and (3) a discrepancy between the different methods on the strength of the ocean sink over the last decade. This living data update documents changes in the methods and data sets used in this new global carbon budget and the progress in understanding of the global carbon cycle compared with previous publications of this data set. The data presented in this work are available at https://doi.org/10.18160/GCP-2022 (Friedlingstein et al., 2022b)