Search for: All records

Creators/Authors contains: "Xie, Jianwen"

Note: When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).
What is a DOI Number?

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

  1. We present a central-peripheral vision-inspired framework (CVP), a simple yet effective multimodal model for spatial reasoning that draws inspiration from the two types of human visual fields – central vision and peripheral vision. Existing approaches primarily rely on unstructured representations, such as point clouds, voxels, or patch features, and inject scene context implicitly via coordinate embeddings. However, this often results in limited spatial reasoning capabilities due to the lack of explicit, high-level structural understanding. To address this limitation, we introduce two complementary components into a Large Multimodal Model-based architecture: target-affinity token, analogous to central vision, that guides the model’s attention toward query-relevant objects; and allocentric grid, akin to peripheral vision, that captures global scene context and spatial arrangements. These components work in tandem to enable structured, context-aware understanding of complex 3D environments. Experiments show that CVP achieves state-of-the-art performance across a range of 3D scene understanding benchmarks. 
    more » « less
    Free, publicly-accessible full text available June 6, 2027
  2. Free, publicly-accessible full text available April 23, 2027
  3. Free, publicly-accessible full text available April 10, 2027
  4. Video understanding in multimodal language models remains limited by context length: models often miss key transition frames and struggle to maintain coherence across long time scales. To address this, we adapt Native Sparse Attention (NSA) to video-language models. Our method, VideoNSA, adapts Qwen2.5-VL through end-to-end training on a 216K video instruction dataset. We employ a hardware-aware hybrid approach to attention, preserving dense attention for text, while employing NSA for video. Compared to token-compression and training-free sparse baselines, VideoNSA achieves competitive performance on long-video understanding, temporal reasoning, and spatial benchmarks. Further ablation analysis reveals four key findings: (1) reliable scaling to 128K tokens; (2) an optimal global–local attention allocation at a fixed budget; (3) taskdependent branch usage patterns; and (4) the learnable combined sparse attention help induce dynamic attention sinks. 
    more » « less
    Free, publicly-accessible full text available April 23, 2027
  5. Free, publicly-accessible full text available April 10, 2027
  6. Free, publicly-accessible full text available April 23, 2027
  7. Free, publicly-accessible full text available April 23, 2027
  8. Free, publicly-accessible full text available April 10, 2027
  9. Free, publicly-accessible full text available May 15, 2027
  10. Machine learning interatomic potentials (MLIPs) can predict energy, force, and stress of materials and enable a wide range of downstream discovery tasks. A key design choice in MLIPs involves the trade-off between invariant and equivariant architectures. Invariant models offer computational efficiency but may not perform as well, especially when predicting high-order outputs. In contrast, equivariant models can capture high-order symmetries, but are computationally expensive. In this work, we propose HIENet, a hybrid invariant-equivariant materials interatomic potential model that integrates both invariant and equivariant message passing layers. We show that HIENet provably satisfies key physical constraints and achieves strong empirical performance across a wide range of benchmarks and downstream discovery tasks. Further evaluations show that HIENet achieves considerable computational speedups compared to prior equivariant models. Finally, additional ablations demonstrate that our hybrid invariant-equivariant approach scales well across model sizes and works with different equivariant model architectures, providing insights into future MLIP designs. 
    more » « less
    Free, publicly-accessible full text available May 1, 2027