More Like this

1. Visual Search Engine for Handwritten and Typeset Math in Lecture Videos and LATEX Notes

   https://doi.org/10.1109/ICFHR-2018.2018.00018  

   Davila, Kenny; Zanibbi, Richard (August 2018, Proc. International Conference on Frontiers in Handwriting Recognition)

   o fill a gap in online educational tools, we are working to support search in lecture videos using formulas from lecture notes and vice versa. We use an existing system to convert single-shot lecture videos to keyframe images that capture whiteboard contents along with the times they appear. We train classifiers for handwritten symbols using the CROHME dataset, and for LATEX symbols using generated images. Symbols detected in video keyframes and LATEX formula images are indexed using Line-of-Sight graphs. For search, we lookup pairs of symbols that can 'see' each other, and connected pairs are merged to identify the largest match within each indexed image. We rank matches using symbol class probabilities and angles between symbol pairs. We demonstrate how our method effectively locates formulas between typeset and handwritten images using a set of linear algebra lectures. By combining our search engine Tangent-V) with temporal keyframe metadata, we are able to navigate to where a query formula in LATEX is first handwritten in a lecture video. Our system is available as open-source. For other domains, only the OCR modules require updating. 

   more » « less
2. Visual Parsing with Query-Driven Global Graph Attention (QD-GGA): Preliminary Results for Handwritten Math Formula Recognition

   https://doi.org/10.1109/CVPRW50498.2020.00293  

   Mahdavi, Mahshad; Sun, Leilei; Zanibbi, Richard (June 2020, Proc. CVPR Workshop on Text and Documents ion the Deep Learning Era)

   We present a new visual parsing method based on convolutional neural networks for handwritten mathematical formulas. The Query-Driven Global Graph Attention (QD- GGA) parsing model employs multi-task learning, and uses a single feature representation for locating, classifying, and relating symbols. First, a Line-Of-Sight (LOS) graph is computed over the handwritten strokes in a formula. Second, class distributions for LOS nodes and edges are obtained using query-specific feature filters (i.e., attention) in a single feed-forward pass. Finally, a Maximum Spanning Tree (MST) is extracted from the weighted graph. Our preliminary results show that this is a promising new approach for visual parsing of handwritten formulas. Our data and source code are publicly available. 

   more » « less
3. ICPR 2020 - Competition on Harvesting Raw Tables from Infographics

   https://doi.org/10.1007/978-3-030-68793-9_27  

   Davila, Kenny; Tensmeyer, Chris; Shekhar, Sumit; Singh, Hrituraj; Setlur, Srirangaraj; Govindaraju, Venu. (February 2021, Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021)

   Del Bimbo, Alberto; Cucchiara, Rita; Sclaroff, Stan; Farinella, Giovanni M; Mei, Tao; Bertini, Marc; Escalante, Hugo J; Vezzani, Roberto. (Ed.)

   This work summarizes the results of the second Competition on Harvesting Raw Tables from Infographics (ICPR 2020 CHART-Infographics). Chart Recognition is difficult and multifaceted, so for this competition we divide the process into the following tasks: Chart Image Classification (Task 1), Text Detection and Recognition (Task 2), Text Role Classification (Task 3), Axis Analysis (Task 4), Legend Analysis (Task 5), Plot Element Detection and Classification (Task 6.a), Data Extraction (Task 6.b), and End-to-End Data Extraction (Task 7). We provided two sets of datasets for training and evaluation of the participant submissions. The first set is based on synthetic charts (Adobe Synth) generated from real data sources using matplotlib. The second one is based on manually annotated charts extracted from the Open Access section of the PubMed Central (UB PMC). More than 25 teams registered out of which 7 submitted results for different tasks of the competition. While results on synthetic data are near perfect at times, the same models still have room to improve when it comes to data extraction from real charts. The data, annotation tools, and evaluation scripts have been publicly released for academic use. 

   more » « less
4. ICDAR 2019 Competition on Harvesting Raw Tables from Infographics (CHART-Infographics)

   https://doi.org/10.1109/ICDAR.2019.00203  

   Davila, Kenny; Kota, Bhargava Urala; Setlur, Srirangaraj; Govindaraju, Venu; Tensmeyer, Christopher; Shekhar, Sumit; Chaudhry, Ritwick (September 2019, 2019 International Conference on Document Analysis and Recognition (ICDAR))

   This work summarizes the results of the first Competition on Harvesting Raw Tables from Infographics (ICDAR 2019 CHART-Infographics). The complex process of automatic chart recognition is divided into multiple tasks for the purpose of this competition, including Chart Image Classification (Task 1), Text Detection and Recognition (Task 2), Text Role Classification (Task 3), Axis Analysis (Task 4), Legend Analysis (Task 5), Plot Element Detection and Classification (Task 6.a), Data Extraction (Task 6.b), and End-to-End Data Extraction (Task 7). We provided a large synthetic training set and evaluated submitted systems using newly proposed metrics on both synthetic charts and manually-annotated real charts taken from scientific literature. A total of 8 groups registered for the competition out of which 5 submitted results for tasks 1-5. The results show that some tasks can be performed highly accurately on synthetic data, but all systems did not perform as well on real world charts. The data, annotation tools, and evaluation scripts have been publicly released for academic use. 

   more » « less
5. Derivation and Validation of an Algorithm to Detect Stroke Using Arm Accelerometry Data

   https://doi.org/10.1161/JAHA.122.028819  

   Messé, Steven R.; Kasner, Scott E.; Cucchiara, Brett L.; McGarvey, Michael L.; Cummings, Stephanie; Acker, Michael A.; Desai, Nimesh; Atluri, Pavan; Wang, Grace J.; Jackson, Benjamin M.; et al (February 2023, Journal of the American Heart Association)

   Background Early diagnosis is essential for effective stroke therapy. Strokes in hospitalized patients are associated with worse outcomes compared with strokes in the community. We derived and validated an algorithm to identify strokes by monitoring upper limb movements in hospitalized patients. Methods and Results A prospective case–control study in hospitalized patients evaluated bilateral arm accelerometry from patients with acute stroke with lateralized weakness and controls without stroke. We derived a stroke classifier algorithm from 123 controls and 77 acute stroke cases and then validated the performance in a separate cohort of 167 controls and 33 acute strokes, measuring false alarm rates in nonstroke controls and time to detection in stroke cases. Faster detection time was associated with more false alarms. With a median false alarm rate among nonstroke controls of 3.6 (interquartile range [IQR], 2.1–5.0) alarms per patient per day, the median time to detection was 15.0 (IQR, 8.0–73.5) minutes. A median false alarm rate of 1.1 (IQR. 0–2.2) per patient per day was associated with a median time to stroke detection of 29.0 (IQR, 11.0–58.0) minutes. There were no differences in algorithm performance for subgroups dichotomized by age, sex, race, handedness, nondominant hemisphere involvement, intensive care unit versus ward, or daytime versus nighttime. Conclusions Arm movement data can be used to detect asymmetry indicative of stroke in hospitalized patients with a low false alarm rate. Additional studies are needed to demonstrate clinical usefulness. 

   more » « less