Carnegie Mellon University, Johns Hopkins University, and New York University created the Project Equity-focused Launch to Empower and Value AGEP Faculty to Thrive in Engineering (ELEVATE) Alliance (National Science Foundation Awards #2149995, #2149798 #2149899 from the Division of Equity for Excellence in STEM in the Directorate for STEM Education) to develop a model to promote the equitable advancement of early career tenure-track engineering faculty from populations of interest to the Alliances for Graduate Education and the Professoriate (AGEP) program. The goal of this AGEP Faculty Career Pathways Alliance Model (FCPAM) is to develop, implement, self-study, and institutionalize a career pathway model that can be adapted for use at other similar institutions for advancing early career engineering faculty who are: African Americans, Hispanic Americans, American Indians, Alaska Natives, Native Hawaiians, and Native Pacific Islanders. This NSF AGEP FCPAM will provide a framework for institutional change at private, highly selective research institutions that will enable all faculty to be members of a collaborative community. Improving the experience of these faculty can lead to increased diversity in the engineering faculty and ultimately result in graduating more engineering students from diverse populations and increasing diversity in the engineering workforce.
The Alliance interventions will focus on three major areas, 1) equity-focused institutional change designed to make structural changes that support the advancement of AGEP faculty, 2) identity-affirming mentorship that acknowledges and provides professional support to AGEP faculty holistically, recognizing all parts of their identity and 3) inclusive professional development that equips all engineering faculty and institutional leaders with skills to implement inclusive practices and equips AGEP faculty for career advancement.
In this paper, we will discuss the process of creating a leadership team to address these focus areas and assess the processes and procedures that currently exist at the three institutions as we begin to institutionalize these change efforts. We provide an overview of the project and efforts to date. We will also present our process for engaging in our initial self-study evaluation and next steps.
more »
« less
Pathways and Structures: Evaluating Systems Changes in an NSF INCLUDES Alliance
In this article, we reflect on our experience applying a framework for evaluating systems change to an evaluation of a statewide West Virginia alliance funded by the National Science Foundation (NSF) to improve the early persistence of rural, first-generation, and other underrepresented minority science, technology, engineering, and mathematics (STEM) students in their programs of study. We begin with a description of the project and then discuss the two pillars around which we have built our evaluation of this project. Next, we present the challenge we confronted (despite the utility of our two pillars) in identifying and analyzing systems change, as well as the literature we consulted as we considered how to address this difficulty. Finally, we describe the framework we applied and examine how it helped us and where we still faced quandaries. Ultimately, this reflection serves two key purposes: (1) to consider a few of the challenges of measuring changes in systems, and (2) to discuss our experience applying one framework to address these issues.
more » « less- NSF-PAR ID:
- 10361234
- Publisher / Repository:
- SAGE Publications
- Date Published:
- Journal Name:
- American Journal of Evaluation
- Volume:
- 43
- Issue:
- 4
- ISSN:
- 1098-2140
- Page Range / eLocation ID:
- p. 632-646
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract Social, political, and cultural complexities observed in environmental justice (EJ) communities require new forms of investigation, science teaching, and communication. Defined broadly, participatory approaches can challenge and change inequity and mistrust in science. Here, we describe Project Harvest and the partnership building and co‐generation of knowledge alongside four EJ communities in Arizona. From 2017 to 2021, Project Harvest centered learning around these communities and the participant experience drove the data sharing practice. The framework of sense‐making is used to analyze how community scientists (CS) are learning within the context of environmental pollution and (in)justice. The environmental health literacy (EHL) framework is applied to document the acquisition of skills that enable protective decision‐making and the capacity of CS to move along the EHL continuum. Using data from surveys, focus groups, and semi‐structured interviews, we are asking how did: (1) Personal connections and local relevancy fuel sense‐making? (2) Data sharing make pollution visible and connect to historical knowledge to either reinforce or modify their existing mental map around pollution? and (3) The co‐creation process build data literacy and a relationship science? Results indicate that due to the program framing, CS personally connected with, and made sense of their data based on use and experience. CS synthesized and connected their pollution history and lived experiences with their data and evaluated contaminant transport. CS saw themselves as part of the process, are taking what they learned and the evidence they helped produce to adopt protective environmental health measures and are applying these skills to new contexts. Here, co‐created science nurtured a new/renewed relationship with science. This science culture rooted in co‐creation, fosters action, trust, and supports ongoing science engagement. The science learning that stems from co‐created efforts can set the pace for social transformation and provide the foundation for structural change.
-
Obeid, I. (Ed.)The Neural Engineering Data Consortium (NEDC) is developing the Temple University Digital Pathology Corpus (TUDP), an open source database of high-resolution images from scanned pathology samples [1], as part of its National Science Foundation-funded Major Research Instrumentation grant titled “MRI: High Performance Digital Pathology Using Big Data and Machine Learning” [2]. The long-term goal of this project is to release one million images. We have currently scanned over 100,000 images and are in the process of annotating breast tissue data for our first official corpus release, v1.0.0. This release contains 3,505 annotated images of breast tissue including 74 patients with cancerous diagnoses (out of a total of 296 patients). In this poster, we will present an analysis of this corpus and discuss the challenges we have faced in efficiently producing high quality annotations of breast tissue. It is well known that state of the art algorithms in machine learning require vast amounts of data. Fields such as speech recognition [3], image recognition [4] and text processing [5] are able to deliver impressive performance with complex deep learning models because they have developed large corpora to support training of extremely high-dimensional models (e.g., billions of parameters). Other fields that do not have access to such data resources must rely on techniques in which existing models can be adapted to new datasets [6]. A preliminary version of this breast corpus release was tested in a pilot study using a baseline machine learning system, ResNet18 [7], that leverages several open-source Python tools. The pilot corpus was divided into three sets: train, development, and evaluation. Portions of these slides were manually annotated [1] using the nine labels in Table 1 [8] to identify five to ten examples of pathological features on each slide. Not every pathological feature is annotated, meaning excluded areas can include focuses particular to these labels that are not used for training. A summary of the number of patches within each label is given in Table 2. To maintain a balanced training set, 1,000 patches of each label were used to train the machine learning model. Throughout all sets, only annotated patches were involved in model development. The performance of this model in identifying all the patches in the evaluation set can be seen in the confusion matrix of classification accuracy in Table 3. The highest performing labels were background, 97% correct identification, and artifact, 76% correct identification. A correlation exists between labels with more than 6,000 development patches and accurate performance on the evaluation set. Additionally, these results indicated a need to further refine the annotation of invasive ductal carcinoma (“indc”), inflammation (“infl”), nonneoplastic features (“nneo”), normal (“norm”) and suspicious (“susp”). This pilot experiment motivated changes to the corpus that will be discussed in detail in this poster presentation. To increase the accuracy of the machine learning model, we modified how we addressed underperforming labels. One common source of error arose with how non-background labels were converted into patches. Large areas of background within other labels were isolated within a patch resulting in connective tissue misrepresenting a non-background label. In response, the annotation overlay margins were revised to exclude benign connective tissue in non-background labels. Corresponding patient reports and supporting immunohistochemical stains further guided annotation reviews. The microscopic diagnoses given by the primary pathologist in these reports detail the pathological findings within each tissue site, but not within each specific slide. The microscopic diagnoses informed revisions specifically targeting annotated regions classified as cancerous, ensuring that the labels “indc” and “dcis” were used only in situations where a micropathologist diagnosed it as such. Further differentiation of cancerous and precancerous labels, as well as the location of their focus on a slide, could be accomplished with supplemental immunohistochemically (IHC) stained slides. When distinguishing whether a focus is a nonneoplastic feature versus a cancerous growth, pathologists employ antigen targeting stains to the tissue in question to confirm the diagnosis. For example, a nonneoplastic feature of usual ductal hyperplasia will display diffuse staining for cytokeratin 5 (CK5) and no diffuse staining for estrogen receptor (ER), while a cancerous growth of ductal carcinoma in situ will have negative or focally positive staining for CK5 and diffuse staining for ER [9]. Many tissue samples contain cancerous and non-cancerous features with morphological overlaps that cause variability between annotators. The informative fields IHC slides provide could play an integral role in machine model pathology diagnostics. Following the revisions made on all the annotations, a second experiment was run using ResNet18. Compared to the pilot study, an increase of model prediction accuracy was seen for the labels indc, infl, nneo, norm, and null. This increase is correlated with an increase in annotated area and annotation accuracy. Model performance in identifying the suspicious label decreased by 25% due to the decrease of 57% in the total annotated area described by this label. A summary of the model performance is given in Table 4, which shows the new prediction accuracy and the absolute change in error rate compared to Table 3. The breast tissue subset we are developing includes 3,505 annotated breast pathology slides from 296 patients. The average size of a scanned SVS file is 363 MB. The annotations are stored in an XML format. A CSV version of the annotation file is also available which provides a flat, or simple, annotation that is easy for machine learning researchers to access and interface to their systems. Each patient is identified by an anonymized medical reference number. Within each patient’s directory, one or more sessions are identified, also anonymized to the first of the month in which the sample was taken. These sessions are broken into groupings of tissue taken on that date (in this case, breast tissue). A deidentified patient report stored as a flat text file is also available. Within these slides there are a total of 16,971 total annotated regions with an average of 4.84 annotations per slide. Among those annotations, 8,035 are non-cancerous (normal, background, null, and artifact,) 6,222 are carcinogenic signs (inflammation, nonneoplastic and suspicious,) and 2,714 are cancerous labels (ductal carcinoma in situ and invasive ductal carcinoma in situ.) The individual patients are split up into three sets: train, development, and evaluation. Of the 74 cancerous patients, 20 were allotted for both the development and evaluation sets, while the remain 34 were allotted for train. The remaining 222 patients were split up to preserve the overall distribution of labels within the corpus. This was done in hope of creating control sets for comparable studies. Overall, the development and evaluation sets each have 80 patients, while the training set has 136 patients. In a related component of this project, slides from the Fox Chase Cancer Center (FCCC) Biosample Repository (https://www.foxchase.org/research/facilities/genetic-research-facilities/biosample-repository -facility) are being digitized in addition to slides provided by Temple University Hospital. This data includes 18 different types of tissue including approximately 38.5% urinary tissue and 16.5% gynecological tissue. These slides and the metadata provided with them are already anonymized and include diagnoses in a spreadsheet with sample and patient ID. We plan to release over 13,000 unannotated slides from the FCCC Corpus simultaneously with v1.0.0 of TUDP. Details of this release will also be discussed in this poster. Few digitally annotated databases of pathology samples like TUDP exist due to the extensive data collection and processing required. The breast corpus subset should be released by November 2021. By December 2021 we should also release the unannotated FCCC data. We are currently annotating urinary tract data as well. We expect to release about 5,600 processed TUH slides in this subset. We have an additional 53,000 unprocessed TUH slides digitized. Corpora of this size will stimulate the development of a new generation of deep learning technology. In clinical settings where resources are limited, an assistive diagnoses model could support pathologists’ workload and even help prioritize suspected cancerous cases. ACKNOWLEDGMENTS This material is supported by the National Science Foundation under grants nos. CNS-1726188 and 1925494. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. REFERENCES [1] N. Shawki et al., “The Temple University Digital Pathology Corpus,” in Signal Processing in Medicine and Biology: Emerging Trends in Research and Applications, 1st ed., I. Obeid, I. Selesnick, and J. Picone, Eds. New York City, New York, USA: Springer, 2020, pp. 67 104. https://www.springer.com/gp/book/9783030368432. [2] J. Picone, T. Farkas, I. Obeid, and Y. Persidsky, “MRI: High Performance Digital Pathology Using Big Data and Machine Learning.” Major Research Instrumentation (MRI), Division of Computer and Network Systems, Award No. 1726188, January 1, 2018 – December 31, 2021. https://www. isip.piconepress.com/projects/nsf_dpath/. [3] A. Gulati et al., “Conformer: Convolution-augmented Transformer for Speech Recognition,” in Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2020, pp. 5036-5040. https://doi.org/10.21437/interspeech.2020-3015. [4] C.-J. Wu et al., “Machine Learning at Facebook: Understanding Inference at the Edge,” in Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA), 2019, pp. 331–344. https://ieeexplore.ieee.org/document/8675201. [5] I. Caswell and B. Liang, “Recent Advances in Google Translate,” Google AI Blog: The latest from Google Research, 2020. [Online]. Available: https://ai.googleblog.com/2020/06/recent-advances-in-google-translate.html. [Accessed: 01-Aug-2021]. [6] V. Khalkhali, N. Shawki, V. Shah, M. Golmohammadi, I. Obeid, and J. Picone, “Low Latency Real-Time Seizure Detection Using Transfer Deep Learning,” in Proceedings of the IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 2021, pp. 1 7. https://www.isip. piconepress.com/publications/conference_proceedings/2021/ieee_spmb/eeg_transfer_learning/. [7] J. Picone, T. Farkas, I. Obeid, and Y. Persidsky, “MRI: High Performance Digital Pathology Using Big Data and Machine Learning,” Philadelphia, Pennsylvania, USA, 2020. https://www.isip.piconepress.com/publications/reports/2020/nsf/mri_dpath/. [8] I. Hunt, S. Husain, J. Simons, I. Obeid, and J. Picone, “Recent Advances in the Temple University Digital Pathology Corpus,” in Proceedings of the IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 2019, pp. 1–4. https://ieeexplore.ieee.org/document/9037859. [9] A. P. Martinez, C. Cohen, K. Z. Hanley, and X. (Bill) Li, “Estrogen Receptor and Cytokeratin 5 Are Reliable Markers to Separate Usual Ductal Hyperplasia From Atypical Ductal Hyperplasia and Low-Grade Ductal Carcinoma In Situ,” Arch. Pathol. Lab. Med., vol. 140, no. 7, pp. 686–689, Apr. 2016. https://doi.org/10.5858/arpa.2015-0238-OA.more » « less
-
more » « less
Abstract For wildlife inhabiting snowy environments, snow properties such as onset date, depth, strength, and distribution can influence many aspects of ecology, including movement, community dynamics, energy expenditure, and forage accessibility. As a result, snow plays a considerable role in individual fitness and ultimately population dynamics, and its evaluation is, therefore, important for comprehensive understanding of ecosystem processes in regions experiencing snow. Such understanding, and particularly study of how wildlife–snow relationships may be changing, grows more urgent as winter processes become less predictable and often more extreme under global climate change. However, studying and monitoring wildlife–snow relationships continue to be challenging because characterizing snow, an inherently complex and constantly changing environmental feature, and identifying, accessing, and applying relevant snow information at appropriate spatial and temporal scales, often require a detailed understanding of physical snow science and technologies that typically lie outside the expertise of wildlife researchers and managers. We argue that thoroughly assessing the role of snow in wildlife ecology requires substantive collaboration between researchers with expertise in each of these two fields, leveraging the discipline‐specific knowledge brought by both wildlife and snow professionals. To facilitate this collaboration and encourage more effective exploration of wildlife–snow questions, we provide a five‐step protocol: (1) identify relevant snow property information; (2) specify spatial, temporal, and informational requirements; (3) build the necessary datasets; (4) implement quality control procedures; and (5) incorporate snow information into wildlife analyses. Additionally, we explore the types of snow information that can be used within this collaborative framework. We illustrate, in the context of two examples, field observations, remote‐sensing datasets, and four example modeling tools that simulate spatiotemporal snow property distributions and, in some cases, evolutions. For each type of snow data, we highlight the collaborative opportunities for wildlife and snow professionals when designing snow data collection efforts, processing snow remote sensing products, producing tailored snow datasets, and applying the resulting snow information in wildlife analyses. We seek to provide a clear path for wildlife professionals to address wildlife–snow questions and improve ecological inference by integrating the best available snow science through collaboration with snow professionals.
-
This paper describes a process that integrates behavioral and decision science methods to design and evaluate interventions to disrupt illicit behaviors. We developed this process by extending a framework used to study systems with uncertain outcomes, where only partial information is observable, and wherein there are multiple participating parties with competing goals. The extended framework that we propose builds from artefactual data collection, thematic analysis, and descriptive analysis, toward predictive modeling and agent-based modeling. We use agent-based modeling to characterize and predict interactions between system participants for the purpose of improving our understanding of interventional targets in a virtual environment before piloting them in the field. We apply our extended framework to an exploratory case study that examines the potential of worker centers as a venue for deploying interventions to address labor exploitation and human trafficking. This case study focuses on reducing wage theft, the most prevalent form of exploitation experienced by day laborers and applies the first three steps of the extended framework. Specifically, the case study makes a preliminary assessment of two types of social interventions designed to disrupt exploitative processes and improve the experiences of day laborers, namely: (1) advocates training day laborers about their workers’ rights and options that they have for addressing wage theft and (2) media campaigns designed to disseminate similar educational messages about workers’ rights and options to address wage theft through broadcast channels. Applying the extended framework to this case study of day laborers at a worker center demonstrates how digital technology could be used to monitor, evaluate, and support collaborations between worker center staff and day laborers. Ideally, these collaborations could be improved to mitigate the risks and costs of wage theft, build trust between worker center stakeholders, and address communication challenges between day laborers and employers, in the context of temporary work. Based on the application of the extended framework to this case study of worker center day laborers, we discuss how next steps in the research framework should prioritize understanding how and why employers make decisions to participate in wage theft and the potential for restorative justice and equity matching as a relationship model for employers and laborers in a well-being economy.more » « less
