Search for: All records

Principled decision-making in continuous state-action spaces is impossible without some assumptions. A common approach is to assume Lipschitz continuity of the Q-function. We show that, unfortunately, this property fails to hold in many typical domains. We propose a new coarse-grained smoothness definition that generalizes the notion of Lipschitz continuity, is more widely applicable, and allows us to compute significantly tighter bounds on Q-functions, leading to improved learning. We provide a theoretical analysis of our new smoothness definition, and discuss its implications and impact on control and exploration in continuous domains. 

Abstract Permafrost underlies approximately one quarter of Northern Hemisphere terrestrial surfaces and contains 25–50% of the global soil carbon (C) pool. Permafrost soils and the C stocks within are vulnerable to ongoing and future projected climate warming. The biogeography of microbial communities inhabiting permafrost has not been examined beyond a small number of sites focused on local-scale variation. Permafrost is different from other soils. Perennially frozen conditions in permafrost dictate that microbial communities do not turn over quickly, thus possibly providing strong linkages to past environments. Thus, the factors structuring the composition and function of microbial communities may differ from patterns observed in other terrestrial environments. Here, we analyzed 133 permafrost metagenomes from North America, Europe, and Asia. Permafrost biodiversity and taxonomic distribution varied in relation to pH, latitude and soil depth. The distribution of genes differed by latitude, soil depth, age, and pH. Genes that were the most highly variable across all sites were associated with energy metabolism and C-assimilation. Specifically, methanogenesis, fermentation, nitrate reduction, and replenishment of citric acid cycle intermediates. This suggests that adaptations to energy acquisition and substrate availability are among some of the strongest selective pressures shaping permafrost microbial communities. The spatial variation in metabolic potential has primed communities for specific biogeochemical processes as soils thaw due to climate change, which could cause regional- to global- scale variation in C and nitrogen processing and greenhouse gas emissions. 

Lakes set in arctic permafrost landscapes can be susceptible to rapid drainage and downstream flood generation. Of many thousands of lakes in northern Alaska, hundreds have been identified as having high drainage potential directly to river systems and 18 such drainage events have been documented since 1955. In 2018 we began monitoring a large lake with high drainage potential as part of a long‐term hydrological observation network designed to evaluate impacts of land use and climate change. In early June 2022, surface water was observed flowing over a 30‐m wide bluff, with active headward erosion of ice‐rich permafrost soils apparent by late June. This overflow point breached rapidly in early July, draining almost the entire lake within 12 h and generating a 191 m³/s flood to a downstream creek. Water level and turbidity sensors and time‐lapse cameras captured this rapid lake‐drainage event at high resolution. A wind‐driven surface seiche and warming waters following ice‐out helped trigger the initial thermomechanical breach. We estimate at least 600 MT of lake sediment was eroded, mobilized, and transported downstream. A flood wave peaking at 42 m³/s arrived 14 h after the initial breach at a river gauge 9‐km downstream. Comparing this event with three other quantified arctic lake‐drainage floods suggests that lake surface area coupled with drainage gradient height can predict outburst flood magnitude. Using this relationship we estimated future flood hazards from the 146 lakes in the Arctic Coastal Plain of northern Alaska (ACP) with high drainage potential, of which 20% are expected to generate outburst floods exceeding 100 m³/s to downstream rivers. This fortunate and detailed drainage‐event observation adds to a growing body of research on the impact of lakes on arctic hydrology, hazard forecasting in a region with an increasing human footprint, and broader processes of landscape evolution in arctic lowlands.

 

A protein roadblock forms when a protein binds DNA and hinders translocation of other DNA binding proteins. These roadblocks can have significant effects on gene expression and regulation as well as DNA binding. Experimental methods for studying the effects of such roadblocks often target endogenous sites or introduce non-variable specific sites into DNAs to create binding sites for artificially introduced protein roadblocks. In this work, we describe a method to create programmable roadblocks using dCas9, a cleavage deficient mutant of the CRISPR effector nuclease Cas9. The programmability allows us to custom design target sites in a synthetic gene intended for in vitro studies. These target sites can be coded with multivalency—in our case, internal restriction sites which can be used in validation studies to verify complete binding of the roadblock. We provide full protocols and sequences and demonstrate how to use the internal restriction sites to verify complete binding of the roadblock. We also provide example results of the effect of DNA roadblocks on the translocation of the restriction endonuclease NdeI, which searches for its cognate site using one dimensional diffusion along DNA. 

Image formation by Fresnel diffraction utilizes both absorption and phase-contrast to measure electron density profiles. The low spatial and spectral coherence requirements allow the technique to be performed with a laser-produced x-ray source coupled with a narrow slit. This makes it an excellent candidate for probing interfaces between materials at extreme conditions, which can only be generated at large-scale laser or pulsed power facilities. Here, we present the results from a proof-of-principle experiment demonstrating an effective ∼2 μm laser-generated source at the OMEGA Laser Facility. This was achieved using slits of 1 × 30  μm 2 and 2 × 40  μm 2 geometry, which were milled into 30 μm thick Ta plates. Combining these slits with a vanadium He-like 5.2 keV source created a 1D imaging system capable of micrometer-scale resolution. The principal obstacles to achieving an effective 1 μm source are the slit tilt and taper—where the use of a tapered slit is necessary to increase the alignment tolerance. We demonstrate an effective source size by imaging a 2 ± 0.2 μm radius tungsten wire. 

Optimistic initialization underpins many theoretically sound exploration schemes in tabular domains; however, in the deep function approximation setting, optimism can quickly disappear if initialized naively. We propose a framework for more effectively incorporating optimistic initialization into reinforcement learning for continuous control. Our approach uses metric information about the state-action space to estimate which transitions are still unexplored, and explicitly maintains the initial Q-value optimism for the corresponding state-action pairs. We also develop methods for efficiently approximating these training objectives, and for incorporating domain knowledge into the optimistic envelope to improve sample efficiency. We empirically evaluate these approaches on a variety of hard exploration problems in continuous control, where our method outperforms existing exploration techniques. 

A fundamental assumption of reinforcement learning in Markov decision processes (MDPs) is that the relevant decision process is, in fact, Markov. However, when MDPs have rich observations, agents typically learn by way of an abstract state representation, and such representations are not guaranteed to preserve the Markov property. We introduce a novel set of conditions and prove that they are sufficient for learning a Markov abstract state representation. We then describe a practical training procedure that combines inverse model estimation and temporal contrastive learning to learn an abstraction that approximately satisfies these conditions. Our novel training objective is compatible with both online and offline training: it does not require a reward signal, but agents can capitalize on reward information when available. We empirically evaluate our approach on a visual gridworld domain and a set of continuous control benchmarks. Our approach learns representations that capture the underlying structure of the domain and lead to improved sample efficiency over state-of-the-art deep reinforcement learning with visual features—often matching or exceeding the performance achieved with hand-designed compact state information.