An official website of the United States government
We propose UniPose, a unified framework for human pose estimation, based on our “Waterfall” Atrous Spatial Pooling architecture, that achieves state-of-art-results on several pose estimation metrics. Current pose estimation methods utilizing standard CNN architectures heavily rely on statistical postprocessing or predefined anchor poses for joint localization. UniPose incorporates contextual segmentation and joint localization to estimate the human pose in a single stage, with high accuracy, without relying on statistical postprocessing methods. The Waterfall module in UniPose leverages the efficiency of progressive filtering in the cascade architecture, while maintaining multiscale fields-of-view comparable to spatial pyramid configurations. Additionally, our method is extended to UniPoseLSTM for multi-frame processing and achieves state-of-theart results for temporal pose estimation in Video. Our results on multiple datasets demonstrate that UniPose, with a ResNet backbone and Waterfall module, is a robust and efficient architecture for pose estimation obtaining state-ofthe-art results in single person pose detection for both single images and videos
more »
« less
Democratization of deep learning for segmenting cartilage from MRIs of human knees: Application to data from the osteoarthritis initiative
Abstract In this study, we aimed to democratize access to convolutional neural networks (CNN) for segmenting cartilage volumes, generating state‐of‐the‐art results for specialized, real‐world applications in hospitals and research. Segmentation of cross‐sectional and/or longitudinal magnetic resonance (MR) images of articular cartilage facilitates both clinical management of joint damage/disease and fundamental research. Manual delineation of such images is a time‐consuming task susceptible to high intra‐ and interoperator variability and prone to errors. Thus, enabling reliable and efficient analyses of MRIs of cartilage requires automated segmentation of cartilage volumes. Two main limitations arise in the development of hospital‐ or population‐specific deep learning (DL) models for image segmentation: specialized knowledge and specialized hardware. We present a relatively easy and accessible implementation of a DL model to automatically segment MRIs of human knees with state‐of‐the‐art accuracy. In representative examples, we trained CNN models in 6‐8 h and obtained results quantitatively comparable to state‐of‐the‐art for every anatomical structure. We established and evaluated our methods using two publicly available MRI data sets originating from the Osteoarthritis Initiative, Stryker Imorphics, and Zuse Institute Berlin (ZIB), as representative test cases. We use Google Colabfor editing and adapting the Python codes and selecting the runtime environment leveraging high‐performance graphical processing units. We designed our solution for novice users to apply to any data set with relatively few adaptations requiring only basic programming skills. To facilitate the adoption of our methods, we provide a complete guideline for using our methods and software, as well as the software tools themselves. Clinical significance: We establish and detail methods that clinical personal can apply to create their own DL models without specialized knowledge of DL nor specialized hardware/infrastructure and obtain results comparable with the state‐of‐the‐art to facilitate both clinical management of joint damage/disease and fundamental research.
more »
« less
- Award ID(s):
- 1653358
- PAR ID:
- 10418853
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Journal of Orthopaedic Research
- Volume:
- 41
- Issue:
- 8
- ISSN:
- 0736-0266
- Page Range / eLocation ID:
- p. 1754-1766
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This paper studied the changing pattern of knee cartilage using 3D knee magnetic resonance (MR) images over a 12-month period. As a pilot study, we focused on the medial tibia compartment of the knee joint. To quantify the thickness of cartilage in this compartment, we utilized two methods: one was measurement through manual segmentation of cartilage on each slice of the 3D MR sequence; the other was measurement through cartilage damage index (CDI), which quantified the thickness on a few informative locations on cartilage. We employed the artificial neural networks (ANNs) to model the changing pattern of cartilage thickness. The input feature space was composed of the thickness information at a cartilage location as well as its neighborhood from baseline year data. The output categories were ‘changed’ and ‘no-change’, based on the thickness difference at the same location between the baseline year and the 12-month follow-up data. Different ANN models were trained by using CDI features and manual segmentation features. Further, for each type of feature, individual models were trained at different subregions of the medial tibia compartment, i.e., the bottom part, the middle part, the upper part, and the whole. Based on the experiment results, we found that CDI features generated better prediction performance than manual segmentation, on both whole medial tibia compartment and any subregion. For CDI, the best performance in term of AUC was obtained using the central CDI locations (AUC = 0.766), while the best performance for manual segmentation was obtained using all slices of the 3D MR sequence (AUC = 0.656). As experiment results showed, the CDI method demonstrated a stronger pattern of cartilage change than the manual segmentation method, which required up to 6-hour manual delineation of all MRI slices. The result should be further validated by extending the experiment to other compartments.more » « less
-
null (Ed.)The task of instance segmentation in videos aims to consistently identify objects at pixel level throughout the entire video sequence. Existing state-of-the-art methods either follow the tracking-bydetection paradigm to employ multi-stage pipelines or directly train a complex deep model to process the entire video clips as 3D volumes. However, these methods are typically slow and resourceconsuming such that they are often limited to offline processing. In this paper, we propose SRNet, a simple and efficient framework for joint segmentation and tracking of object instances in videos. The key to achieving both high efficiency and accuracy in our framework is to formulate the instance segmentation and tracking problem into a unified spatial-relation learning task where each pixel in the current frame relates to its object center, and each object center relates to its location in the previous frame. This unified learning framework allows our framework to perform join instance segmentation and tracking through a single stage while maintaining low overheads among different learning tasks. Our proposed framework can handle two different task settings and demonstrates comparable performance with state-of-the-art methods on two different benchmarks while running significantly faster.more » « less
-
Today’s generative models can synthesize magnetic resonance images (MRIs) of the brain at specific ages. However, such models can neither map the aging process longitudinally within subjects, nor accommodate its variability across subjects. Such approaches also cannot predict anatomic features of aging in ways that can be validated retrospectively or trusted prospectively. We introduce a three-dimensional hybrid ControlNet + diffusion model that uses the baseline T1-weighted MRIs of healthy adults to predict individual neuroanatomic aging trajectories, as reflected by follow-up MRIs. The approach captures individual anatomical changes with an average predicted voxelwise intensity error of 15% and structural similarity index of 93%. Unlike methods relying on qualitative validation, our approach quantifies the fidelity of prospective MRI synthesis using FreeSurfer volumetrics. Because brain atrophy reflects risk for Alzheimer’s disease (AD), our model’s ability to generate individual-specific prospective MRIs suggests its clinical potential to assist AD risk estimation.more » « less
-
Abstract Background We aimed to determine if composite structural measures of knee osteoarthritis (KOA) progression on magnetic resonance (MR) imaging can predict the radiographic onset of accelerated knee osteoarthritis. Methods We used data from a nested case-control study among participants from the Osteoarthritis Initiative without radiographic KOA at baseline. Participants were separated into three groups based on radiographic disease progression over 4 years: 1) accelerated (Kellgren-Lawrence grades [KL] 0/1 to 3/4), 2) typical (increase in KL, excluding accelerated osteoarthritis), or 3) no KOA (no change in KL). We assessed tibiofemoral cartilage damage (four regions: medial/lateral tibia/femur), bone marrow lesion (BML) volume (four regions: medial/lateral tibia/femur), and whole knee effusion-synovitis volume on 3 T MR images with semi-automated programs. We calculated two MR-based composite scores. Cumulative damage was the sum of standardized cartilage damage. Disease activity was the sum of standardized volumes of effusion-synovitis and BMLs. We focused on annual images from 2 years before to 2 years after radiographic onset (or a matched time for those without knee osteoarthritis). To determine between group differences in the composite metrics at all time points, we used generalized linear mixed models with group (3 levels) and time (up to 5 levels). For our prognostic analysis, we used multinomial logistic regression models to determine if one-year worsening in each composite metric change associated with future accelerated knee osteoarthritis (odds ratios [OR] based on units of 1 standard deviation of change). Results Prior to disease onset, the accelerated KOA group had greater average disease activity compared to the typical and no KOA groups and this persisted up to 2 years after disease onset. During a pre-radiographic disease period, the odds of developing accelerated KOA were greater in people with worsening disease activity [versus typical KOA OR (95% confidence interval [CI]): 1.58 (1.08 to 2.33); versus no KOA: 2.39 (1.55 to 3.71)] or cumulative damage [versus typical KOA: 1.69 (1.14 to 2.51); versus no KOA: 2.11 (1.41 to 3.16)]. Conclusions MR-based disease activity and cumulative damage metrics may be prognostic markers to help identify people at risk for accelerated onset and progression of knee osteoarthritis.more » « less
