Search for: All records

Abstract. We present 4Diff, a 3D-aware diffusion model addressing the exo-to-ego viewpoint translation task—generating first-person (egocentric) view images from the corresponding third-person (exocentric) images. Building on the diffusion model’s ability to generate photorealistic images, we propose a transformer-based diffusion model that incorporates geometry priors through two mechanisms: (i) egocentric point cloud rasterization and (ii) 3D-aware rotary cross-attention. Egocentric point cloud rasterization converts the input exocentric image into an egocentric layout, which is subsequently used by a diffusion image transformer. As a component of the diffusion transformer’s denoiser block, the 3D-aware rotary cross-attention further incorporates 3D information and semantic features from the source exocentric view. Our 4Diff achieves state-of-the-art results on the challenging and diverse Ego-Exo4D multiview dataset and exhibits robust generalization to novel environments not encountered during training. Our code, processed data, and pretrained models are publicly available at https://klauscc.github.io/4diff. 

The increasing popularity of deep learning models has created new opportunities for developing AI-based recommender systems. Designing recommender systems using deep neural networks requires careful architecture design, and further optimization demands extensive co-design efforts on jointly optimizing model architecture and hardware. Design automation, such as Automated Machine Learning (AutoML), is necessary to fully exploit the potential of recommender model design, including model choices and model-hardware co-design strategies. We introduce a novel paradigm that utilizes weight sharing to explore abundant solution spaces. Our paradigm creates a large supernet to search for optimal architectures and co-design strategies to address the challenges of data multi-modality and heterogeneity in the recommendation domain. From a model perspective, the supernet includes a variety of operators, dense connectivity, and dimension search options. From a co-design perspective, it encompasses versatile Processing-In-Memory (PIM) configurations to produce hardware-efficient models. Our solution space’s scale, heterogeneity, and complexity pose several challenges, which we address by proposing various techniques for training and evaluating the supernet. Our crafted models show promising results on three Click-Through Rates (CTR) prediction benchmarks, outperforming both manually designed and AutoML-crafted models with state-of-the-art performance when focusing solely on architecture search. From a co-design perspective, we achieve 2 × FLOPs efficiency, 1.8 × energy efficiency, and 1.5 × performance improvements in recommender models. 

Cretaceous eolian deposits provide evidence of variations in the tropical-subtropical atmospheric circulation under greenhouse conditions. However, the misinterpretation of many such deposits as fluvial or deltaic originally hindered precise paleoclimatic reconstructions. Here we report a newly identified Early Cretaceous desert in the Hami Basin, China, which helps understand spatial-temporal variations in aridity and atmospheric circulations within central East Asia during the Early Cretaceous. The Liushuquan Formation is composed of >300-m-thick eolian deposits interpreted as an intermontane erg environment. Paleocurrent indicators within the straight-crested dunes of the Liushuquan Formation yield a mean trend of 101.3° (± 10.1°, 1 standard deviation) throughout the formation, consistent with near-surface westerly winds. Paleo-atmospheric circulation superimposed on topographic effects led to widespread eolianite accumulation during the Early Cretaceous. Combined with the spatiotemporal changes in desert distributions and prevailing surface wind patterns in East Asia, these observations are consistent with the migration of the subtropical high-pressure belt during the Early Cretaceous. We propose the following paleo-atmospheric model: (1) During the late Berriasian−Valanginian, the subtropical high belt drifted southward and northward over shorter time scales within the spatial domain of the paleo-Ordos Basin, then shifted southward at least past the Ordos Basin; (2) until the late Hauterivian−Barremian, the subtropical high-pressure zone was primarily located between the northwestern Tarim Basin and the Ordos Basin; and (3) a significant southward shift of the subtropical high-pressure zone occurred during the Aptian−Albian.