Search for: All records

Abstract Qualitative coding, or content analysis, is more than just labeling text: it is a reflexive interpretive practice that shapes research questions, refines theoretical insights, and illuminates subtle social dynamics. As large language models (LLMs) become increasingly adept at nuanced language tasks, questions arise about whether—and how—they can assist in large-scale coding without eroding the interpretive depth that distinguishes qualitative analysis from traditional machine learning and other quantitative approaches to natural language processing. In this paper, we present a hybrid approach that preserves hermeneutic value while incorporating LLMs to scale the application of codes to large data sets that are impractical for manual coding. Our workflow retains the traditional cycle of codebook development and refinement, adding an iterative step to adapt definitions for machine comprehension, before ultimately replacing manual with automated text categorization. We demonstrate how to rewrite code descriptions for LLM-interpretation, as well as how structured prompts and prompting the model to explain its coding decisions (chain-of-thought) can substantially improve fidelity. Empirically, our case study of socio-historical codes highlights the promise of frontier AI language models to reliably interpret paragraph-long passages representative of a humanistic study. Throughout, we emphasize ethical and practical considerations, preserving space for critical reflection, and the ongoing need for human researchers’ interpretive leadership. These strategies can guide both traditional and computational scholars aiming to harness automation effectively and responsibly—maintaining the creative, reflexive rigor of qualitative coding while capitalizing on the efficiency afforded by LLMs. 

Abstract The Institutional Grammar (IG) is a rigorous tool for analyzing the laws and policies governing nonprofit organizations; however, its use was limited due to the time-consuming nature of hand-coding. We introduce an advance in Natural Language Processing using a semantic role labeling (SRL) classifier that reliably codes rules governing and guiding nonprofit organizations. This paper provides guidance for how to hand-code using the IG, preprocess text for machine learning, and demonstrates the SRL classifier for automated IG coding. We then compare the hand-coding to the SRL coding to demonstrate its accuracy. The advances in machine learning now make it feasible to utilize the IG for nonprofit research questions focused on inter-organizational collaborations, government contracts, federated nonprofit organizational compliance, and nonprofit governance, among others. An added benefit is that the IG is adaptable for different languages, thus enabling cross-national comparative research. By providing examples throughout the paper, we demonstrate how to use the IG and the SRL classifier to address research questions of interest to nonprofit scholars. 

ABSTRACT Institutional arrangements that guide collective action between entities create benefits and burdens for collaborating entities and can encourage cooperation or create coordination dilemmas. There is an abundance of research in public policy, public administration, and nonprofit management on cross‐sector alliances, co‐production, and collaborative networks. We contribute to advancing this research by introducing a methodological approach that combines two text‐based methods: institutional network analysis and cost–benefit analysis. We utilize the Institutional Grammar to code policy documents that govern relationships between actors. The coded text is then used to identify Networks of Prescribed Interactions to analyze institutional relationships between policy actors. We then utilize the coded text in a cost–benefit analysis to assess benefit and burden distributive effects. This integrated methodological framework provides researchers with a tool to elucidate both the institutional patterns of interaction and distributive implications embedded in policy documents, revealing insights that single‐method approaches cannot capture. We then utilize the coded text in a cost–benefit analysis to assess benefit and burden distributive effects. This integrated methodological framework provides researchers with a tool to elucidate both the institutional patterns of interaction and distributive implications embedded in policy documents, revealing insights that single‐method approaches cannot capture. To demonstrate the utility of this integrated approach, we examine the policy design of two nonprofit open‐source software (OSS) incubation programs with contrasting characteristics: the Apache Software Foundation (ASF) and the Open Source Geospatial Foundation (OSGeo). We select these cases because: (1) they are co‐production alliances and have policy documents that articulate support for collective action; (2) their policy documents and group discussions are open access, creating an opportunity to advance text‐based policy analysis methods; and (3) they represent juxtaposed examples of high and low risk for collaboration settings, thereby providing two illustrative cases of the combined network and cost–benefit text‐based methodological approach. The network analysis finds that ASF policies, as a high‐risk setting, emphasize bonding structures, particularly higher reciprocity, which creates a context for cooperation. OSGeo, a low‐risk setting, has policies creating a context for bridging structures, evident in high brokerage efficiency, to facilitate coordination. The cost–benefit analysis finds that ASF policies balance the distribution of costs and benefits between ASF and projects, while in OSGeo, projects bear both costs and benefits. These findings demonstrate that the combination of network and cost–benefit analysis is an effective tool for utilizing text to compare policy designs. 

Open source software (OSS) underpins modern software infrastructure, yet many projects struggle with long- term sustainability. We introduce OSSPREY, an AI-powered platform that can predict the sustainability of any GitHub- hosted project. OSSPREY collects longitudinal socio-technical data, such as: commits, issues, and contributor interactions, and uses a transformer-based model to generate month-by-month sustainability forecasts. When project downturns are detected, it recommends evidence-based interventions drawn from published software engineering studies. OSSPREY integrates scraping, forecasting, and actionable guidance into an interactive dash- board, enabling maintainers to monitor project health, anticipate decline, and respond with targeted strategies. By connecting real- time project data with research-backed insights, OSSPREY offers a practical tool for sustaining OSS projects at scale. The codebase is linked to the project website at: https: //oss-prey.github.io/OSSPREY-Website/ The screencast is available at: https://www.youtube.com/ watch?v=N7a0v4hPylU 

We study how open source communities describe participation and control through version controlled governance documents. Using a corpus of 710 projects with paired snapshots, we parse text into actors, rules, actions, and objects, then group them and measure change with entropy for evenness, richness for diversity, and Jensen Shannon divergence for drift. Projects define more roles and more actions over time, and these are distributed more evenly, while the composition of rules remains stable. These findings indicate that governance grows by expanding and balancing categories of participation without major shifts in prescriptive force. The analysis provides a reproducible baseline for evaluating whether future AI mediated workflows concentrate or redistribute authority. 

Ethical concerns around AI have increased emphasis on model auditing and reporting requirements. We thoroughly review the current state of governance and evaluation prac- tices to identify specific challenges to responsible AI devel- opment in OSS. We then analyze OSS projects to understand if model evaluation is associated with safety assessments, through documentation of limitations, biases, and other risks. Our analysis of 7902 Hugging Face projects found that while risk documentation is strongly associated with evaluation practices, high performers from the platform’s largest com- petitive leaderboard (N=789) were less accountable. Recog- nizing these delicate tensions from performance incentives may guide providers in revisiting the objectives of evaluation and legal scholars in formulating platform interventions and policies that balance innovation and responsibility. 

Large Language Models (LLMs) have become pivotal in reshaping the world by enabling advanced natural language processing tasks such as document analysis, content generation, and conversational assistance. Their ability to process and generate human-like text has unlocked unprecedented opportunities across different domains such as healthcare, education, finance, and more. However, commercial LLM platforms face several limitations, including data privacy concerns, context size restrictions, lack of parameter configurability, and limited evaluation capabilities. These shortcomings hinder their effectiveness, particularly in scenarios involving sensitive information, large-scale document analysis, or the need for customized output. This underscores the need for a tool that combines the power of LLMs with enhanced privacy, flexibility, and usability. To address these challenges, we present EvidenceBot, a local, Retrieval-Augmented Generation (RAG)-based solution designed to overcome the limitations of commercial LLM platforms. Evidence-Bot enables secure and efficient processing of large document sets through its privacy-preserving RAG pipeline, which extracts and appends only the most relevant text chunks as context for queries. The tool allows users to experiment with hyperparameter configurations, optimizing model responses for specific tasks, and includes an evaluation module to assess LLM performance against ground truths using semantic and similarity-based metrics. By offering enhanced privacy, customization, and evaluation capabilities, EvidenceBot bridges critical gaps in the LLM ecosystem, providing a versatile resource for individuals and organizations seeking to leverage LLMs effectively. 

In the rapidly evolving domain of software engineering (SE), Large Language Models (LLMs) are increasingly leveraged to automate developer support. Open source LLMs have grown competitive with pro- prietary models such as GPT-4 and Claude-3, without the associated financial and accessibility constraints. This study investigates whether state of the art open source LLMs including Solar-10.7B, CodeLlama-7B, Mistral-7B, Qwen2-7B, StarCoder2-7B, and LLaMA3-8B can generate responses to technical queries that align with those crafted by human experts. Leveraging retrieval augmented generation (RAG) and targeted fine tuning, we evaluate these models across critical performance dimen- sions, such as semantic alignment and contextual fluency. Our results show that Solar-10.7B, particularly when paired with RAG and fine tun- ing, most closely replicates expert level responses, o!ering a scalable and cost e!ective alternative to commercial models. This vision paper high- lights the potential of open-source LLMs to enable robust and accessible AI-powered developer assistance in software engineering. 

Online communities rely on effective governance for success, and volunteer moderators are crucial for ensuring such governance. Despite their significance, much remains to be explored in understanding the relationship between community governance processes and moderators’ psychological experiences. To bridge this gap, we conducted an online survey with over 600 moderators from Reddit communities, exploring the link between different governance strategies and moderators’ needs and motivations. Our investigation reveals a contrast to conventional views on democratic governance within online communities. While participatory processes are associated with higher levels of perceived fairness, they are also linked with reduced feelings of community belonging and lower levels of institutional acceptance among moderators. Our findings challenge the assumption that greater democratic involvement unequivocally leads to positive community outcomes, suggesting instead that more centralized governance approaches can also positively affect moderators’ psychological well-being and, by extension, community cohesion and effectiveness. 

Many have criticized the centralized and unaccountable governance of prominent online social platforms, leading to renewed interest in platform governance that incorporates multiple centers of power. Decentralization of power can arise horizontally, through parallel communities, each with local administration, and vertically, through multiple hierarchies of overlapping jurisdiction. Drawing from literature on federalism and polycentricity in analogous offline institutions, we scrutinize the landscape of existing platforms through the lens of multi-level governance. Our analysis describes how online platforms incorporate varying forms and degrees of decentralized governance. In particular, we propose a framework that characterizes the general design space and the various ways that middle levels of governance vary in how they can interact with a centralized governance system above and end users below. This focus provides a starting point for new lines of inquiry between platform- and community-governance scholarship. By engaging themes of decentralization, hierarchy, power, and responsibility, while discussing concrete examples, we connect designers and theorists of online spaces.