More Like this

1. EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition

   https://doi.org/10.1016/j.neucom.2025.129781  

   Abdelkawy, Ahmed; Ali, Asem; Farag, Aly (February 2025, Neurocomputing)

   Existing multimodal-based human action recognition approaches are computationally intensive, limiting their deployment in real-time applications. In this work, we present a novel and efficient pose-driven attention-guided multimodal network (EPAM-Net) for action recognition in videos. Specifically, we propose eXpand temporal Shift (X-ShiftNet) convolutional architectures for RGB and pose streams to capture spatio-temporal features from RGB videos and their skeleton sequences. The X-ShiftNet tackles the high computational cost of the 3D CNNs by integrating the Temporal Shift Module (TSM) into an efficient 2D CNN, enabling efficient spatiotemporal learning. Then skeleton features are utilized to guide the visual network stream, focusing on keyframes and their salient spatial regions using the proposed spatial–temporal attention block. Finally, the predictions of the two streams are fused for final classification. The experimental results show that our method, with a significant reduction in floating-point operations (FLOPs), outperforms and competes with the state-of-the-art methods on NTU RGB-D 60, NTU RGB-D 120, PKU-MMD, and Toyota SmartHome datasets. The proposed EPAM-Net provides up to a 72.8x reduction in FLOPs and up to a 48.6x reduction in the number of network parameters. The code will be available at https://github.com/ahmed-nady/Multimodal-ActionRecognition. 

   more » « less
2. Holistic 3D Human and Scene Mesh Estimation from Single View Images.

   Weng, Zhenzhen; Yeung, Serena (January 2021, IEEE Xplore digital library)

   null (Ed.)

   The 3D world limits the human body pose and the hu- man body pose conveys information about the surrounding objects. Indeed, from a single image of a person placed in an indoor scene, we as humans are adept at resolving am- biguities of the human pose and room layout through our knowledge of the physical laws and prior perception of the plausible object and human poses. However, few computer vision models fully leverage this fact. In this work, we pro- pose a holistically trainable model that perceives the 3D scene from a single RGB image, estimates the camera pose and the room layout, and reconstructs both human body and object meshes. By imposing a set of comprehensive and sophisticated losses on all aspects of the estimations, we show that our model outperforms existing human body mesh methods and indoor scene reconstruction methods. To the best of our knowledge, this is the first model that outputs both object and human predictions at the mesh level, and performs joint optimization on the scene and human poses. 

   more » « less
3. Mapping Spatial Patterns to Energetic Benefits in Groups of Flow-coupled Swimmers

   https://doi.org/10.1101/2024.02.15.580536  

   Heydari, Sina; Hang, Haotian; Kanso, Eva (February 2024, bioRxiv)

   Abstract The coordinated motion of animal groups through fluids is thought to reduce the cost of locomotion to individuals in the group. However, the connection between the spatial patterns observed in collectively moving animals and the energetic benefits at each position within the group remains unclear. To address this knowledge gap, we study the spontaneous emergence of cohesive formations in groups of fish, modeled as flapping foils, all heading in the same direction. We show in pairwise formations and with increasing group size that (1) in side-by-side arrangements, the reciprocal nature of flow coupling results in an equal distribution of energy re-quirements among all members, with reduction in cost of locomotion for swimmers flapping inphase but an increase in cost for swimmers flapping antiphase, and (2) in inline arrangements, flow coupling is non-reciprocal for all flapping phase, with energetic savings in favor of trailing swimmers, but only up to a finite number of swimmers, beyond which school cohesion and energetic benefits are lost at once. We explain these findings mechanistically and we provide efficient diagnostic tools for identifying locations in the wake of single and multiple swimmers that offer op-portunities for hydrodynamic benefits to aspiring followers. Our results imply a connection between the resources generated by flow physics and social traits that influence greedy and cooperative group behavior. 

   more » « less
4. PSMNet: Position-aware Stereo Merging Network for Room Layout Estimation

   Haiyan Wang, Will Hutchcroft (June 2022, IEEE Conference on Computer Vision and Pattern Recognition (CVPR))

   In this paper, we propose a new deep learning-based method for estimating room layout given a pair of 360◦ panoramas. Our system, called Position-aware Stereo Merging Network or PSMNet, is an end-to-end joint layout-pose estimator. PSMNet consists of a Stereo Pano Pose (SP2) transformer and a novel Cross-Perspective Projection (CP2) layer. The stereo-view SP2 transformer is used to implicitly infer correspondences between views, and can handle noisy poses. The pose-aware CP2 layer is designed to render features from the adjacent view to the anchor (reference) view, in order to perform view fusion and estimate the visible layout. Our experiments and analysis validate our method, which significantly outperforms the state-of-the-art layout estimators, especially for large and complex room spaces. 

   more » « less
5. GeniusRoute: A New Analog Routing Paradigm Using Generative Neural Network Guidance

   https://doi.org/10.1109/ICCAD45719.2019.8942164  

   Zhu, Keren; Liu, Mingjie; Lin, Yibo; Xu, Biying; Li, Shaolan; Tang, Xiyuan; Sun, Nan; Pan, David Z. (November 2019, 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD))

   Due to sensitive layout-dependent effects and varied performance metrics, analog routing automation for performance-driven layout synthesis is difficult to generalize. Existing research has proposed a number of heuristic layout constraints targeting specific performance metrics. However, previous frameworks fail to automatically combine routing with human intelligence. This paper proposes a novel, fully automated, analog routing paradigm that leverages machine learning to provide routing guidance, mimicking the sophisticated manual layout approaches. Experiments show that the proposed methodology obtains significant improvements over existing techniques and achieves competitive performance to manual layouts while being capable of generalizing to circuits of different functionality. 

   more » « less