Surfacing Equity Issues in Large Computing Courses with Peer-Ranked, Demographically-Labeled Student Feedback

courses face persistent diversity, equity, and inclusion issues [42,85]. In part due to the growing demand for computing skills in the workforce, CIS enrollment numbers have surged recently, straining the capacity of instructors to scale teaching [61]. Despite their popularity, CIS courses continue to face challenges retaining and supporting diverse students in both high school [24] and college [82,85,91]. This results in the loss of diverse potential contributors to computing felds [61] and raises social justice issues around who can access and engage with computing communities [46,74,75].

One way to to try to teach more equitably is by sourcing feedback from students and responding to it [16]. Feedback is especially critical to an equitable learning environment because students from minoritized groups face unique challenges that, if left unaddressed, can pose serious impediments to science and technology learning [20]. Instructors in higher education have used student feedback as a way to monitor and improve teaching equality, especially in distance and remote learning environments [43]. However, it is not enough to simply be made aware that these challenges exist: To help turn student feedback into action, instructors need context regarding their students' lived experiences to understand how challenges afect diferent students [17,54].

Student feedback tools in large (100-500+ student) courses must be scalable. Prior CSCW works have examined massive open online courses (MOOCs), looking at student motivations and retention [89], the impact of a reputation system on the student experience using forums [13], and how matching students across locations helped students to earn higher grades [48]. But in contrast to MOOCs, remote courses typically have more synchronous interactions and feedback mechanisms between instructors and students. Scalable feedback in synchronous courses involves ensuring convenience for students to share feedback [29,44], and convenience for instructors and teaching assistants (TAs) to collect, analyze, and discuss the feedback [43,69]. In addition to this requirement, we argue that student feedback for equity-oriented goals must also provide the context to help instructors and TAs consider feedback within the context of students' lived experiences while also ensuring students' privacy.

Context is important to support equity-oriented goals, but existing student feedback mechanisms lack the context to connect feedback to lived experiences. At the scale of hundreds of students to a single teaching team consisting typically of one instructor and a few TAs, a teaching team typically cannot respond to all feedback. As a result, commonly used electronic response systems, such as anonymous online surveys sent to students during the term, often lack context about lived experience [12]. This loss of context results in less actionable feedback as it obscures perspectives of minoritized groups as they become lost amongst the majority of perspectives which are typically from students of dominant groups [54].

Contextualizing feedback can come in tension with protecting student privacy, another critical aspect for equity-oriented feedback. Students from minoritized groups are often most at risk when their information is exposed without their informed consent. Feedback methods that are interpersonal and conversational, such as conversations between a student and an instructor after a lecture or with a teaching assistant (TA) during ofce hours, are common within large computing courses. While these methods provide context by revealing students' identity, they also privilege students who are more willing and able to speak up, such as white and Asian men with prior programming experience [29]. Interpersonal techniques can be especially problematic for students of minoritized groups due to a lack of anonymity potentially introducing stereotype threat [79,88] and social-desirability biases [28,33]. Student feedback for equity-oriented goals must also ensure students' privacy when they share feedback.

To explore the design of a student feedback tool that supports equity-oriented goals by 1) being scalable, 2) providing context, and 3) ensuring student privacy, we designed StudentAmp.

StudentAmp contextualizes student feedback on challenges in their life with demographics and peer

Surfacing Inequities in Computing Courses with Contextualized Student Feedback 65:3 perspectives from other students. Using StudentAmp, students self-report challenges that interfered with their learning as well as demographic information (e.g. gender, ethnicity, prior programming knowledge). Students then consider random pairs of challenges their classmates/peers shared and determine which challenge they would consider more disruptive. StudentAmp aggregates students' meta-feedback responses to produce a ranking of perceived challenge disruptiveness. StudentAmp then uses this data to produce a report for an instructor detailing challenges students reported, contextualizing challenges with demographic information as well as ranking them according to student perceptions of disruptiveness.

To evaluate the efects of using StudentAmp to collect and report student feedback contextualized by demographic information and perceptions from classmates, we conducted a formative study with teaching teams and students of fve large remote computing courses (163 -628 students/course). We considered the following research questions:

(1) What do students share about challenges interfering with their learning? (2) How do students perceive the values and risks of sharing information on challenges they face, contextualized with demographics and peer-perceptions? (3) How do teaching teams of large computing courses use diferent types of information to contextualize students challenges for equity-oriented interpretations?

We found that students considered the privacy of themselves and others when sharing feedback on challenges that were often about their lives beyond computing courses. Seeing anonymous peers' challenges also helped students empathize and develop a sense of belonging with peers. Instructors used demographic data to connect challenges to student experiences by situating challenges in lived experiences that may difer from dominant norms, while fnding data on peer perspectives questionable and unreliable. We interpreted these fndings as design trade-ofs between contextualizing feedback with demographic data to inform stakeholders of inequities at scale and ensuring the privacy and well-being of students.

This paper makes the following three contributions:

(1) A large scale thematic analysis of 810 challenges that 604 students faced while learning computing in large remote courses during the COVID-19 pandemic; (2) An artifact (StudentAmp) that is a design exploration into how contextualizing student feedback may support more equitable learning experiences in large, remote courses; and (3) A rich qualitative investigation of (a) Students' experiences sharing feedback through StudentAmp and viewing the feedback of their peers, and (b) Teaching teams' experiences using StudentAmp to better understand inequities in their courses by interpreting contextualized student feedback.

Equity Involves Understanding Experiences of Minoritized Groups

We framed equitable learning as ensuring students from diverse backgrounds can successfully access and engage with a learning experience to realize their dignity and potential. Within the context of computing education, understanding students' diverse backgrounds involves considering intersectionality [21], or how diferent aspects of students' identities intersect and interact. These aspects of identities include students' ethnicities, genders, disabilities (physical, mental, social), preparatory privilege, current situation outside of the course (e.g. familial or fnancial responsibilities), and past educational experiences (whether they are a transfer student, frst generation). Learners are complex individuals beyond a single demographic label.

When considering equity, we must consider not only students, but also the structures of society that students exist in. We establish this framing of equity upon upon Structuration Theory, which defnes a recursive process where society and its structures shape the activity of individuals and individuals shape and condition the structures of society [14,47]. Understanding equity involves not just individuals involved, but also the context of the economic, social, cultural, and political conditions of the time and place [27,60]. Equity has a social justice goal where corrective measures must adjust for aggregate harm from social inequalities [67]. As a result, understanding equity involves considering how learners are situated within complex environments that they also shape.

Within computing courses, improving equity would require not just improving access to computing education, but also supporting successful participation and achievement by diverse students learning computing [50]. Structural and systemic inequities embedded in and around computing courses can manifest as barriers to participation (e.g. unconscious bias of instructors excluding students of color from successful participation [70]), afect students' sense of belonging and identity (e.g. instructional materials promoting gender bias [57]), and exacerbate existing disparities in privilege (e.g. students cannot synchronously engage with instructors and other classmates because of timezone diferences, work commitments, or familial responsibilities) [50]. Inequities arise when structures and norms fail to include or serve students of minoritized groups. Addressing inequities often involves interventions that support the needs of specifc groups, such as a one hour social-belonging intervention to support the long term career, mental health, and community building for Black students [7,84].

For this paper, we referred to minoritized as a descriptor of identity groups that are typically not dominant within computing communities in the United States (US). Dominant groups are positively privileged [87], unstigmatized [72], and generally favored by the institutions of society [53], particularly within social, economic, political, and educational systems [23]. For the context of college computer and information science programs in the US, we characterized dominant groups as including white and Asian men who started college shortly after high school (not transfer students), do not have disabilities, have little or no fnancial or familial responsibilities, have English fuency, and have at least one parent who completed a four year college degree. Minoritized groups, then, are groups that are not positively privileged or favored and often stigmatized. In our context of study, minoritized groups include students who are women, non-binary, African-American/Black, Hispanic/Latinx, Native American/Indigenous, Pacifc Islander, transfer students, not fuent in English, and/or frst-generation, as well as students who have disabilities and/or have fnancial or familial responsibilities. While some may consider minoritized groups a small proportion of the population, these groups can actually make up a large proportion of society while still being minoritized by systemic injustices. Systemic cultures and norms tend to favor dominant groups and disadvantage minoritized groups.

Perspective Taking to Beter Understand Students' Situations

To understand others' situations, humans rely (at least partially) on empathy. Empathy is a multidimensional construct, a set of interrelated yet distinct social behaviors and abilities that enable understanding of other peoples' unique contexts [22,71,76]. According to some models, there are two major kinds of empathy: afective empathy, which involves responding with one's own emotion to another person's mental or emotional state; and cognitive empathy, which involves the ability to understand another person's mental state [22]. For this investigation, we focused specifcally on promoting perspective-taking behavior, which is a facet of cognitive empathy involving adopting others' points of view [22].

Perspective taking as a means of empathizing can be useful tool in understanding others' situations and needs. For instance, in design contexts, designers often use some form of perspective taking to try and better understand the needs of diferent groups of stakeholders. Prior work suggests that encouraging perspective taking through the use of personas or cognitive walkthroughs can help promote better understandings of minoritized groups [32,59], including people with diferent genders [10,80], cultures [1], socioeconomic statuses [58], and abilities [5,62].

Supporting proper perspective-taking behavior can be challenging, especially in educational contexts. While there remains comparatively little work on promoting empathy in traditional programming courses, prior work in the area of HCI and software design education suggests that perspective taking can be difcult to teach and to learn [32,63,64,68]. This is especially true in higher education computing contexts, which tend to be dominated by young, cognitively and physically high-performing students who may lack exposure to perspectives and viewpoints that are very diferent than their own [51]. Poorly executed perspective-taking activities may also lead to stereotyping, or making erroneous assumptions about a particular individual based solely on some limited information about them, such as their demographics. Stereotyping is a particular danger when asking people from dominant groups to perspective-take with people in minoritized or less contextually dominant groups [4,8].

Stereotyping is an innate human behavior and cannot be done away with entirely [83]. For instance, if no particular traits about users are specifed, software designers practicing perspective taking may fall back on implicit assumptions that a user is of a contextually dominant race, gender, age, culture, and class, who is heterosexual, afuent, comfortable with technology, and not disabled [19]. However, prior work suggests that providing enough rich contextual information about the target person's identities and behaviors can preclude some of the harmful efects of stereotyping [38]. Providing more information about a person's experiences and identities also can reduce tendencies toward single-axis analysis [19] which can erase the lived experiences of those with intersectional identities. To have the best chances of perspective taking being efective, comprehensive, and benefcial instead of harmful, providing more information about a person can support more holistic understandings of their unique situation.

Theory of Action to Guide Data Interpretation

Becoming aware of challenges is only a frst (but critical) step towards addressing them. That is to say that showing somebody information will not necessarily translate to action. To scafold this connection between data and action, we used Theory of Action, a framework that helps educators develop evidence-based stories that explains the specifc changes they intend to make to improve teaching and learning [11].

We drew upon Theory of Action (ToA) to connect instructors' interpretations of data from StudentAmp to action [11,15,41]. ToA relates individual actions to systemic functioning by articulating the underlying logic of work and starting assumptions about how and why actions will lead to desired outcomes [15,41]. In ToA, actions involve information that stakeholders fnd valuable within their societal structures and can use to afect power dynamics [15]. While originally derived from studies of individual and organizational learning [2,3], educational policymakers and administrators have used ToA to make changes to improve teaching and learning [11,41].

Ongoing development and communication of a ToA can help instructors improve teaching and learning. Most of the work that uses ToA as a guiding framework to improve teaching and learning has primarily focused on schools and school districts teaching primary/elementary and secondary education (e.g. [41,45]). It is typically an iterative process where leaders look at data to understand students' learning experiences, as well as refect on how teachers' instruction afects student learning and how school principals' practices afect teachers' instruction [11,41]. For ToA to make changes that would improve teaching and learning, leaders must articulate ToA and reform plans in terms that are compelling and understandable to multiple stakeholders and lay framework for ongoing "reform conversation" [41].

While leadership in primary and secondary educational institutions in the United Stated tends to be more centralized to school and district levels, post-secondary institutions (e.g. colleges and universities) tend to aford individual teachers (professors) more autonomy over their own classes [9,35]. Leveraging this, we re-framed ToA to remove the principals and instead have teachers/instructors (e.g. professors, lecturers) as the leaders. This new framework positions instructors within the leadership role of 1) looking at data to understand students' learning experiences, 2) refecting on how instruction afects student learning, and 3) identifying how the context surrounding the course afects instruction. This context afecting learning is broad and can include departmental policies (e.g. grade being used for acceptance into competitive major) and current events (e.g. global health emergency, political unrest).

To successfully use ToA to improve a course involves having course instructors serve as stewards who continuously develop, communicate, and advocate for actions. Honig et al. 2010 saw stewardship as critical to the ongoing process of reform with ToA [41]. They identifed tasks that stewards must take which we adapted from a school/district level to a course level: Ongoing development of a theory of action for the transformation of course; communication with others to help them understand the theory of action, including strategies used and underlying rationale for these strategies; and strategic brokering of external resources and relationships to support the overall course transformation process.

We framed StudentAmp as a tool to provide data that course instructors could use to inform the creation, iteration, and application of Theories of Action to improve their course. We scafold the implementation of StudentAmp in courses within the context of creating Theories of Action (defned in [11]).

Critical Self-Reflexivity: Acknowledging Researchers' Positionality

This research required a reduction of people to the responses they were willing to share, so we acknowledge our assumptions and values in this section. By doing so, we follow critical approaches to quantitative methods which require researchers "to engage in critical self-refexivity as a necessary frst step for the long journey of deracializing statistics" [30]. As part of this process, we defne assumptions and commitments that were the foundation of this research.

Firstly, we recognize the power structures and heterogeneity of people within diferent roles. Direct stakeholders in this research included teaching teams (including faculty members leading the teaching of a course and teaching assistants (TAs) supporting the teaching) and students involved in the course. Even within these groups, there were diferences. Faculty members leading instruction ranged from tenured research-track faculty who had worked for years at the institution to teachingtrack lecturers with comparatively less teaching experience. The TAs were all undergraduate students, but their experiences with their respective courses ranged from never having taught or taken it to having years of previous experience taking and teaching the course. Full or part-time students attending these courses may or may not have been accepted to their major (admissions to CIS majors is very competitive and not guaranteed as part of admission to the university). Most students were enrolled to take the course, but some may have had listener status where they were not taking it for ofcial credit. Some transferred from other higher education institutions (e.g. two-year institutions) with diferent norms, while others came directly from high school. A common theme across all stakeholders: The data we collect is a partial and biased lens into their experiences in a select few courses as part of a much larger educational experience.

Secondly, we acknowledge the tensions between labeling people in data, the intersectionality of people's identities (students in particular), and ensuring privacy. Intersectionality denotes the various ways in which ethnicity and gender (and other demographic labels) interact to shape the peoples' lived experiences [21]. Prior work has found that simplistic labeling of people can harm minoritized groups in particular. Labels of demographics (e.g. ethnicity, gender) academic experience (e.g. year in school, major, transfer or not), and lived experience (e.g. disabilities, familial language) are overly-simplistic. Furthermore, we needed to balance the nuance of the labels we select between how representative they were to diverse individuals and how anonymous they were such that instructors could not map responses back to individuals or small groups of students. Despite these risks, we believed that instructors could still use these labels in such a way to help stakeholders contextualize relationships between challenges and intersectional groups of people. Our perspectives align with the notion that "race is a measure of a relationship -not an inalterable trait" [90].

Finally, we acknowledge that models are always wrong in that they never fully refect the complex phenomena we want them to represent, but they can be designed such that they are useful in informing stakeholders of hidden challenges. We framed the work we did as producing simplifed models of the complex phenomena of inequity in classes. We do not believe that in itself models will help, but they can support conversations, interactions, and interventions that address the systemic issues we sought to bring to light [16]. The objective of this work was to help instructors identify equity issues in their class, and that is a frst of many steps in making learning experiences more equitable and just.

Student

View: Sharing challenges, demographics, perspectives on other challenges. For this study, we designed StudentAmp's student view with the intention of enabling a student to be able to share feedback within a few minutes. Students could access StudentAmp from any modern web browser (e.g. Firefox, Chrome, Safari). Figure 1 shows an example of the StudentAmp interface as it appeared to students. Students frst shared "the biggest challenge in [their] life getting in the way of this class, " with helper text which prompted students to think beyond the scope of the class. Text also appeared which encouraged students to share more ("Keep writing so others understand your challenge!") if their response was < 100 characters and more if their response was ≥ 160 characters ("You wrote quite a bit! Consider condensing your writing so others can read it quickly."). From our pilot testing, we found that a message of 100-160 characters (approximately the maximum length of a tweet on the social media platform Twitter) represented sufcient description for another student or instructor to understand a challenge response without being too burdensome to read.

After sharing their challenges, students had the opportunity to self-report demographics, as shown in Figure 1, step 2. We based StudentAmp's demographic questions on factors which prior work found to be impactful to students' learning experiences, including prior programming experience, whether they were a transfer student [49], whether they were frst-generation, whether their familial language is the same as the language the course is taught in (English), gender, ethnicity, and physical, mental, or social disability status. While we required answers for all multiple choice demographics questions, each question included an option for "(prefer not to answer). " If students had previously flled out demographic questions (e.g. in a previous feedback session), StudentAmp populated these questions with the student's prior responses. The demographics questions and options were as follows: (1) How many programming courses have you previously completed? 0 (before this term, I've never taken a programming course); to a small extent; 2: to some extent; 3: to a moderate extent; 4: to a great extent; 5: to a very great extent; I'm not sure; (prefer not to disclose). [select one] (9) Rate to what extent a mental or social disorder hinders your learning experience. 0: not at all; 1: to a small extent; 2: to some extent; 3: to a moderate extent; 4: to a great extent; 5: to a very great extent; I'm not sure; (prefer not to disclose). [select one] After sharing their challenges and demographics, students fnally shared meta-feedback on their classmates' responses, as shown in Figure 1.3. In this phase, StudentAmp randomly selected two challenges classmates had reported and asked students to imagine they had these two challenges, then to select the one that they imagined would be more disruptive to learning. To support data integrity of the meta-feedback, students could skip any responses. The meta-feedback pairwise comparison process was repeated 2-8 times depending on the class size (2 ≤ 2 * ( ) ≤ 8).

Instructor

View: Designing for instructor-led data exploration. Once students shared feedback, StudentAmp presented teaching teams with a report on student feedback, as shown in Figure 2. We designed StudentAmp to augment instructors' domain knowledge related to the course and their students by enabling exploration of contextualized feedback data. StudentAmp enabled this data exploration by 1) informing instructors of challenges, which student groups they afected, and how severe students perceived them to be and 2) supporting situated annotations through labels and notes so instructors could review previous fndings.

As mentioned previously, instructors within our study each created a section for their course which students joined via a unique 6 character code. Each time an instructor wished to use StudentAmp to gather feedback, they created new feedback session. Once students shared feedback via StudentAmp instructors viewed the results, as shown in Figure 2. Teaching teams could review this data to identify how certain types of challenges disproportionately afected certain groups. Instructors and teaching teams could browse reported challenges, sorted by disrupt score (Fig. 2e), identifying trends and patterns. They could then create labels and assign them to challenges (Fig. 2f). We designed StudentAmp's labels to help teaching teams organize and prioritize feedback to better identify trends within and across feedback sessions. Similar to GitHub labels [31], teaching teams could defne any labels they wanted, then assign one or many labels to any responses in any feedback session, similar to a tagging system. While the use of labels did require teaching manually labeling individual feedback (e.g. we did not provide automated labeling), it also enabled fltering of responses to use demographics charts (Fig. 2c) to explore how challenges afected demographic groups. In our study, we saw that this helped teaching teams understand which types of challenges disproportionately afected diferent groups of students across the nine demographic features we collected (enumerated in section 3.3.1).

In StudentAmp, we aimed to support benefcial and efective perspective taking for instructors. Teaching teams could look at challenges (unfltered or fltered by label) and use demographic information associated with individual challenges (Fig. 2d) to perspective take. To help instructors better understand the nuanced ways that diferent challenges afected diferent students, we provided demographic information alongside each challenge to promote more informed perspective taking through the addition of richer contextual information. By doing so, we hoped to avoid The instructor view also included each challenge that included the selected label(s). Each challenge was contextualized with demographics for minoritized groups that students identified with (d), disrupt score (e), and labels that the teaching team assigned to that challenge (f). Teaching teams could also share collaborative notes (g), which have prompting based on our Theory of Action.

stereotyping by encouraging instructors to see their students as unique individuals with many diferent kinds of identities and contexts, rather than defaulting to the assumption of an "average" (likely dominant) student if no information was given. However, we also had to balance this against the need to protect student privacy and preserve anonymity, which we discuss further in our design considerations.

Throughout this entire process, teaching teams could use StudentAmp's collaborative notes (Fig. 2g) to review Theory of Action principles and also share notes with other members of the teaching team. Notes were open-ended text felds shared by all teaching teams. Each feedback session had a separate notes section from which instructors could write notes about their fndings within the report. To help guide teaching teams according to our Theory of Action, we included prompting above the notes. This prompting asked teaching teams to consider 1) What's going on in students' learning experiences?, 2) How does what instructors are doing (or not doing) afect learning experiences? and 3) What factors external to the course help or hinder students' learning experiences? Which students? It then asks students to fll in the blanks as many times as possible: "If I/we __, then the course will change by __, so that students who are __ will be able to __. " 3.3.3 Design of Disrupt Score. We designed StudentAmp's disrupt score with an intention to draw attention to challenges that were more disruptive to students' learning, rather than those that were simply more frequent or relatable to the majority of students. Prior work [20] and our pilot testing identifed that students of minoritized groups reported challenges that were often unique from what their peers shared. As a result, we designed StudentAmp to support equitable feedback processes by organizing and presenting feedback in a way that went beyond showing the most common challenges. The disrupt score provided a quantifcation metric intended to represent how disruptive to students' learning some challenges were compared to other challenges.

In its most literal sense, the disrupt score was the net number of times a random student within the course decided a challenge was more disruptive to learning than another randomly selected challenge. It was based on a pairwise comparison of random challenges, as shown in Figure 1.3 (see also the description of meta-feedback above). We based this strategy on the Copeland method of pairwise scoring to determine ranked voting [18,73]. Each challenge began with a score of 0. Each time a student shared meta-feedback (described in 3.3.1), the challenge they selected had its disrupt score incremented by one (+1). The challenge they did not select correspondingly had its disrupt score decremented by one(-1). No score changes occurred if a student chose to skip a meta-feedback comparison. These scores were then aggregated and used to track challenges in the instructor view in addition to being shown alongside each challenge (Fig. 2e).

A key assumption to the disrupt score was that students were able to perspective take and consider the disruptiveness of challenges they may not even have. Our initial pilot testing with students thinking aloud as they considered challenges found that students were more likely to select the challenge they most associated with having. As a result, for the study results we report in this paper, we adjusted StudentAmp's design by adding more information in prompts and button text to explicitly encourage consideration of disruptiveness rather than readability. As we report and discuss in the following sections, think-alouds with interviewed students suggested that students conceptualized the meta-feedback process in diferent ways (section 5.2.4), which resulted in several teaching teams having trouble interpreting disrupt scores (section 5.3.3).

STUDY DESIGN: DEPLOYED STUDENTAMP AND CONDUCTED INTERVIEWS 4.1 Context: Five large, remote, computing courses during a pandemic

To answer our research questions, we conducted a formative evaluation where fve large computing courses used StudentAmp over the course of an 11 week term and we surveyed and interviewed professors, TAs, and some students twice throughout the term. An institutional review board (IRB) approved this study prior to any data collection.

We recruited culturally competent [86] instructors teaching large computing courses (100+ students). We chose to directly contact instructors who had previously demonstrated cultural competence [86] for participation in the study, as evidenced by their prior research eforts or participation in seminars on anti-racism. We focused on culturally-competent professors to avoid interpretations of student feedback and demographics focused on innate ability, such as certain people or groups having the "geek gene" and being more suited to computing [55,65]. We selected large courses to better ensure anonymity and to have a scale at which analyzing individual feedback of an entire course would become time-consuming and challenging. To avoid potentially unproductive and harmful uses of StudentAmp, we chose to exclude instructors who had previously stated beliefs about demographic groups' interest and abilities in computing [65].

Of the eight professors we reached out to, fve professors teaching large computing courses participated in this study and used StudentAmp. Table 1 provides an overview of those fve courses and how they used StudentAmp. Courses all took place within the same term, during which a global COVID-19 pandemic and a global reckoning with racial injustice were both ongoing issues. Because the university was shut down to in-person learning, all courses in this study were taught remotely, with students located in time zones all over the world. Given courses B, D, and E were introductory courses with no prerequisite requirements, some students in the study were frst year college students who had yet to have an in-person college experience.

The fve courses in our study were from the two departments of the same research university. This public research university in the United States was located in a major city with signifcant presence of large technology companies. All participants from this study (professors, TAs, and students) taught at or attended this university. We interpreted this study context to be Western, Educated, Industrialized, Rich, and Demographic (a.k.a. WEIRD, [37]). While most people around the world are not from WEIRD societies [36], many computing education contexts in the United States tend to be WEIRD societies [82,91].

Courses A, B, and C were diferent courses ofered within the computer science (CS) department. Based on data collected earlier in the same academic year that this study occurred, the department has 1,668 enrolled undergraduates. The CS department reported 31% of their students as female and 69% of undergraduates as male (only binary gender was collected on this survey); 8% of students as under-represented minority/URM (African American, American Indian/Alaska Native, Hawaiian/Pacifc Islander and Latinx/Hispanic) and 75% as non-URM; 20% as frst-generation (none of their parents completed four year college degrees); and 17% as international.

Courses D and E were the same course taught by diferent teaching teams and from the information science department, a separate department from the CS department. Based on enrollment data collected the term after this study, the information science department had 526 undergraduates. The information science department reported 43% of undergraduates as female and 57% as male (only binary gender collected). The department reported 45% as Asian, 12% as URM, 16% as white, 28% unknown. Whether a student was Hispanic/Latinx was reported separately, with 4% (23) reporting as Hispanic/Latinx.

In addition to using StudentAmp, teaching teams relied on other tools and methods to collect student feedback for various purposes. Other feedback tools included feedback after assignments (e.g. to fnd out how long an assignment took, P-B), during lecture (e.g. "to ascertain skill acquisition, " P-D), during the middle of the quarter (e.g. mid-term feedback conducted with an instructional consultant, P-A, P-C), and at the end of the term. They had previously used feedback to "respond to small conveniences that students requested" (P-C). They also invited students to reach out to them directly through email or similar mediums, but P-E noted how students from dominant groups tended to speak up more through these channels:

P-E: "students that have taken a bunch of programming classes and already done this stuf...they're the ones who speak up, who talk, engage [...] and this is all tied to race and gender. " experience. Professors of all fve courses reported as white or Asian men with no physical, mental, or social disabilities. Two professors (for courses A and B) were teaching their courses for the frst time but had experience teaching related courses; the other three had taught that same course multiple times before. All fve professors had been teaching courses remotely for at least the two terms prior to the study. Of the 26 teaching assistants (TAs) who had access to or reviewed StudentAmp responses, 17 responded to a survey to report their their demographics. For ethnicity, 1 TA reported as Hispanic/Latinx, 12 as Asian, and 4 as white (non-Hispanic). For gender, 1 reported as non-binary, 9 as women, and 8 as men. Three TAs reported mental or social disabilities, such as anxiety. All but one TA who responded had previously either taken the course or an equivalent one to the course they were serving as a TA for. That one TA who had no prior experience as a student or TA for the course they were teaching had previously served as TA for the course B professor (P-B) for other courses in the past. Taken together, we can say that TAs were predominantly white or Asian, identifed as a diversity of genders, and generally had prior experience with the course material as a student and/or a TA.

Data for RQ1: 810 challenges shared with StudentAmp.

To understand what challenged students shared, we analyzed challenges shared by students with StudentAmp. In total, 604 unique students shared a total of 810 challenges across the fve courses through StudentAmp over the duration of the 11 week term. The Responses column in Table 1 shows the number of responses in the feedback sessions across the fve courses that used StudentAmp. We included incomplete responses because those responses were only incomplete because those students did not provide meta-feedback. 4.2.2 Data for RQ2: Rounds of interviews with 5 students of minoritized groups to understand perceptions. To understand factors that may have impacted what students shared with their instructors through StudentAmp, we conducted two rounds of semi-structured interviews with students. We recruited students who used StudentAmp and indicated interest in conducting follow-up interviews for compensation ($50 for two 1-1.5 hr interviews, which was slightly above minimum wage in the area at the time). We conducted interviews remotely, recording video (including screen share) and audio with the consent of students.

We interviewed fve students from minoritized groups. Of the 234 students who indicated potential interest in a follow-up interview, we identifed 39 who were from minoritized groups (as evidenced by their ethnicity, gender) and/or reported a unique challenge. We contacted those 39 students by email and ended up interviewing all fve students who replied. These fve students came from three courses (three from course A, one from C, one from D). Three students were Asian women, one was a Hispanic/Latinx woman, and one was a white non-binary personfoot_3 . Two were third year undergraduates studying majors related to computer and information sciences (CIS), two were frst year undergraduates interested (but not yet enrolled) in CIS majors, and one was second year Master's student studying information science. While taking this course, interviewed students reported other commitments including submitting more than 100 job applications, moving physical locations, having familial responsibilities, and taking almost double the recommended course load.

We collected data via two rounds of retrospective think-aloud style interviews [25] to understand how students interpreted the prompts and how they decided to share what they did.

The frst round of interviews occurred within the frst fve weeks of the term, after students had shared feedback using StudentAmp at least once. Students answered questions about their experiences learning computing and expectations about using StudentAmp, and then shared their screen as they walked through their prior usage of StudentAmp (three steps in Fig. 1), reviewing and refecting on their previous responses and the context surrounding them. We asked them to interpret what each page was asking them to do and also about their perceived risks and benefts for sharing information at each page. We closed the frst interview by asking students to refect.

The second round of interviews occurred during the fnal two weeks of the term and included a sorting activity to have students consider additional challenges they did not share as well feedback on an example instructor view (Fig. 2). These interviews asked students to refect on their course experience. We then had students look at all the challenges they shared over the course of term (2-3 challenges per students), asking them to identify potential trends, explain how they decided to share these challenges, and consider how peers potentially seeing their reported challenges afected their decisions to share.We then showed students the StudentAmp instructor view to get their perspectives on the utility and risks of contextualized feedback.

Data for RQ3: Rounds of interviews with 5 teaching teams to understand use of StudentAmp.

To understand how teaching teams used contextualized student feedback for equity-oriented interpretation, we conducted semi-structured interviews with teaching teams who used StudentAmp.

Prior to or during the frst two weeks of the course, we helped members of the courses' teaching teams (professors and TAs) set up their StudentAmp accounts online so they could collect and analyze feedback from their students. We also asked professors to fll out a survey to share their prior experiences with student feedback, the course they were teaching, and demographic information. TAs flled out a similar survey at the end of the term. To understand how teaching teams of large computing courses used diferent types of information to contextualize students challenges for equity-oriented interpretations, we conducted three interviews with members of the teaching teams of the fve courses in our study: One interview with professors individually, followed by two group interviews with the entire teaching team of a course. The interviews with individual professors took place either before the term began or within the frst four weeks of the course. In these interviews, we asked professors to share about the course they were teaching, how they intended to collect feedback from students about various aspects of the course (whether that method was StudentAmp or not), and about their personal defnitions of equity in educational spaces (see Table 1).

The two group interviews were spaced out across the duration of the course, with one occurring towards the beginning of the term (after at least one feedback session with StudentAmp had been completed) and one at the end. We began the frst group interviews by explaining the features of the StudentAmp interface (Fig. 2c,d,e,g), allowing time for the teaching team to ask questions as needed. We then engaged in a process following our Theory of Action (Section 2.3) in which the teaching team tried to understand students' learning experiences using StudentAmp, considered how their teaching practices or external factors might be afecting student learning, and then proposing changes to the course and articulating how those changes might afect students of diferent groups, identities, or experiences. We asked one member of the teaching team to share their screen during this process, so participants could have a single shared view to discuss. After ten minutes of unstructured exploration of students' responses, if teaching teams were not already doing so, we prompted them to begin identifying broader patterns and trends they saw among student responses, create labels (Fig. 2a), and add labels to challenges (Fig 2f). After a few more minutes of labeling (the exact duration of which depended on each teaching team's level of discussion), we prompted teams to consider the demographic information provided at the top of the StudentAmp interface (Fig 2c) and explore any potential interactions of demographics with the labels they had created to explore how groups of similar challenges might disproportionately afect certain groups of students. The second group interviews followed a similar process, with the main diference being that the student responses under review were from subsequent StudentAmp feedback sessions.

Interviews were conducted remotely with the Zoom video conferencing tool, which enabled synchronous video and audio conversation, messaging, and recording. We choose to use this tool because all fve teaching teams were familiar with it as they used it in their remote teaching. We recorded video and audio for all interviews. All members of the teaching team consented to the inclusion of their audio and visual recordings, as well as their survey data in this study. Participants were also compensated at a rate of $15/hr (approximately minimum wage for the area).

RQ1: Students shared challenges beyond the scope of the course

In total, 604 unique students shared a total of 810 challenges across the fve courses through StudentAmp over the duration of the 11 week term. To better understand what students shared, we conducted an inductive thematic analysis and a subsequent round of qualitative coding using themes from the initial analysis. Three researchers participated in the qualitative analysis:

• The frst author, a critical data studies and computing education researcher with seven years of research experience in data equity in computing education. The frst author had expertise in designing interactions with data in educational contexts and mixed methods, having previously taught high school and college courses on introductory computer science and data science. He led the design of StudentAmp and the evaluation. • The second author, a computing education researcher with seven years of research experience in HCI and design methods, including four years researching computing education within that space and a year of teaching experience in higher education computing contexts. The second author had expertise in qualitative methods and led the analysis of student-reported challenges. • The third author, a computing education researcher with eleven years of middle and high school teaching experience, and two years of educational research in computing-related contexts.

First, all three researchers participated in collaborative afnity diagramming of 100 randomly selected challenges to inductively generate initial themes with a sensitizing concept ( [6,66]) of types of challenges. We used these themes as the basis of our code set. All three researchers collaboratively coded 40 (5%) randomly sampled challenges with the initial code set, discussing discrepancies and iteratively refning the code set and code defnitions as needed. As can be seen in many of the below quotes, even though we asked students to report the single biggest challenge they were facing in the class, students often reported multiple, often interwoven, challenges. As a result, we allowed for multiple codes per challenge during the qualitative coding efort. Our goal with this coding efort was to achieve consensus, so all codes applied to a challenge had to be agreed upon by all coders.

After that, two researchers (the second and third authors) continued collaboratively coding another 160 (15%) challenges to ensure both of them had similar interpretations of the code set defnitions and to address any confusions that arose. Finally, the two researchers divided the rest of the data and each qualitatively coded half of the remaining student-reported challenges. Once they fnished their respective analyses, the two researchers asynchronously verifed each others' code applications, marking any instances of disagreement. Finally, the two researchers met synchronously to discuss and come to consensus on the codes applied to the fnal 610 student-reported challenges, discussing interpretations and eventually achieving full agreement on all codes.

Table 2 shows our code set, comprised of the major themes that arose from our analysis and one "Other" code that was applied when a challenge was too unclear to code or when it otherwise did not fall into a coded category. These 16 types of challenges in Table 2 represent diferent learning difculties that students conveyed to instructors through StudentAmp. For this analysis, we adhere to Hammer and Berland's perspective on qualitative coding [34], treating the results of our coding efort as organizations of claims about data rather than quantitative data in and of itself. As a result, we do not report specifc code frequencies, instead focusing on representative descriptions of the themes observed within our data. In the following section, IDs preceding quotes indicate the speaker's role within the class (S for student), the course in which the student was enrolled (see Table 1), and a randomly generated number unique to each student within the course.

Even though StudentAmp's instructor view shows student-reported challenges alongside some demographic information about the student who wrote it, for the purposes of this paper, we choose not to report demographic information of the speaker for each individual quote. Instead, we report the demographics or our participants in aggregate, to illustrate the diversity of perspectives and experiences represented by the data while still preserving our participants' anonymity and reducing the risk of community or peer re-identifcation. The following subsection contains quotes from 21 diferent students to illustrate the kinds of challenges students reported through StudentAmp. 20 of the 21 students who provided these quotes identifed as belonging to at least one minoritized group, and often several. Of these 21 studentsfoot_4 , at the time of the study:

• 10 identifed as women, 10 as men, and 1 declined to provide gender information.

• 12 identifed as Asian, 7 as white, 2 as Hispanic/Latinx, 1 as Black/African, and 1 as Pacifc Islander. • 3 reported taking their frst programming course.

• 4 reported attending another institution prior to their current one (transfer students).

• 5 reported as being frst-generation college students.

• 6 reported that their family spoke a language at home than was diferent from the one used in the course (English). • 6 students reported that they were currently working part-time, and 2 students full-time. 10 students were actively job-searching (e.g. applying to jobs, attending interviews). 5 students were neither working nor job-searching. • 12 students reported that they did not have a physical/bodily disorder that hindered their learning experience (0 on a scale of 0-5). 8 students reported that they had physical/bodily disorders which hindered their learning to a minor extent (1-2 on scale), and 1 to a severe extent (4-5 on scale). • 12 students reported that they did not have a mental or social disorder that hindered their learning experience (0 on a scale of 0-5). 2 students reported that they had mental or social

Other

Challenges that were listed as "N/A" or "nothing", or that did not contain sufcient data to interpret

disorders which hindered their learning to a minor extent (1-2 on scale), 5 to a moderate extent (3 on scale), and 3 to a severe extent (4-5 on scale).

The quotes presented below have been edited as little as possible to preserve authenticity. When clarifcations or minor edits for anonymity were necessary, or when some less relevant parts of the quote were removed for length reasons, we designate any edits with square brackets.

We found the types of challenges students reported to fall into six broad categories (see the second-from-the-left column of Table 2 for an overview). S-A-102: "The structure of this class because we simultaneously learn stuf for the assessment while learning the stuf for the following week forcing me to sacrifce one for the other. "

Other students who reported Course structure codes faced challenges with managing the course's required virtual learning tools or adapting to the instructor's pedagogical style.

When students reported challenges that contained Course content codes, they often mentioned the stress that came from trying to learn computing topics:

S-E-28: "I have never learned coding/data analysis ever in my life, things are just intimidating. IDK this class is STRESSING ME OUT. "

Many students who reported this challenge also reported not having much prior experience with computing, or who hadn't programmed in a long time.

Remote learning challenges were often reported by students who disliked the virtual format of classes, which was mandated by the university in response to the ongoing COVID-19 pandemic:

S-E-6: "The biggest challenge is just the general lack of structure that is inherent in online classes, regardless of how well the instructor organizes the course. "

Some students felt that virtual classes were not as conducive to learning as in-person classes, or that they did not feel like they got as much out of remote classes: S-C-109: "I think the biggest challenge will truly just be the online nature of life right now. Screen fatigue is a big issue for me, & especially knowing that a programming class will require large amounts of screen time after class is a bit daunting. [...] I am worried about feeling intellectually gratifed just because of Zoom fatigue. "

Other students who reported challenges containing Remote learning mentioned issues with poor Internet connections that made it challenging for them to attend virtual classes and difculties connecting with peers and teaching staf. Overall, challenges coded as Course structure, Course content, and Remote learning codes were likely similar to the kinds of feedback instructors might get with traditional feedback mechanisms (surveys, teaching evaluations, etc.).

Non-academic roles include familial and job commitments.

A third higher-level theme of challenges which surfaced during our analysis was that of non-academic roles and responsibilities, represented by the Home & family and Job codes.

Students who reported Home & family challenges often described difculties focusing on coursework in their current environments, which often co-occurred with Remote learning codes. Sometimes, students simply mentioned that it was difcult for them to focus in their home environments, leading to them having to re-watch lectures or take extra time reading course materials. Other students wrote about their roles as caretakers of other family members, which took time away from their own responsibilities: S-B-18: "I have a sibling that is disabled and another sibling that just started kindergarten. Because of this, I have to help my parents with making sure they attend their classes and do their homework, which is time consuming and also takes time out of other responsibilities around the house, plus work. "

Other household stressors, such as sick pets or siblings, also afected students' abilities to focus, whether due to stress about their well-being or having to provide transportation to medical appointments.

S-E-27: "My dog is getting eye surgery today, because of suspected cancer that caused Gluacoma. We don't know if the cancer is malignant yet but it's hard knowing that each day could be his last. :( so far our luck has been pretty bad but I really hope that the other tumors are benign like the one on his stomach. [...]" Many students worked jobs or internships during the quarter, or were actively job searching, as represented by the Job code. Often, this challenge was discussed in terms of time constraints, which sometimes made it difcult for students to engage with instruction or fnd time to complete their assigned work. Several students mentioned they were working full-time or part-time jobs alongside their full-time course loads. Other students described their roles as primary providers for their families: S-A-71: "I am the only one supporting my family economically, so they depend on me working and getting money for our dependancies. "

Sometimes, the job environment or tasks themselves contributed to overall student stress: S-B-29: "My job. I work at a homeless shelter. I only work two shifts a week, but I deal with a lot of very high stress situations (fghts, 911 calls, suicidal ideation, sexual assaults, mental health crises, etc). Balancing school stress and job stress can often be difcult. It has been harder recently as I am regularly exposed to Covid positive individuals, so the likelihood of me catching Covid is very high. "

Both Home & family and Job codes represent roles and responsibilities in students' broader lives that placed demands upon their time and available physical, mental, and emotional resources.

5.1.5

Well-being included physical and mental well-being. The ffth higher-level category of challenges students reported was that of personal well-being getting in the way of their learning, as is the case with the Physical health, Mental health, and Isolation codes. Students who mentioned Physical health sometimes described bouts of illness that caused them to fall behind in their courses or not feel well enough to do coursework. S-A-92: "I had the worst case of mono for about the frst three weeks of class, and now I am trying to catch up and relearn the basics of those frst three weeks. I am having a difcult quarter" Other Physical health codes involved complications related to remote learning, such as sleep schedules being disrupted by having to attend synchronous classes in the middle of the night, or having physical symptoms from virtual learning through screens.

Students who reported Mental health challenges often wrote about stress, anxiety, and depression. S-B-41: "Depression and anxiety, the pressure everyday" These kinds of challenges often co-occurred with reports of heavy course loads, difcult course content, or low engagement with others due to remote learning settings.

Somewhat similarly, Isolation codes had to do with students' socio-emotional well-being, and especially a lack of meaningful interactions with others.

S-A-35: "Being alone and lonely doing CS" Challenges that contained Isolation codes were often reported in conjunction with statements about the COVID-19 pandemic (due to quarantines, lockdowns, etc.) and remote learning.

S-D-37: "Being online is very isolating and does not allow for as much connection between students and instructors. "

All three these types of challenges -Physical health, Mental health, and Isolation -have to do with students' inner well-being, and health is an important prerequisite for efective learning.

5.1.6 Self-regulation involved motivation and time management. Finally, a sixth category of challenges students reported had to do with their own current self-regulation capacities and abilities. These were represented by the Motivation and Time management codes.

When students wrote about challenges that we coded as Motivation, they mentioned struggles with procrastination, distractions, focus, and feeling capable of completing coursework to their own standards.

S-C-205: "I have trouble fnding the motivation to do school work nowadays. " Oftentimes, Motivation codes co-occurred with Remote learning or Isolation codes, when students pointed out that their current environments were making it more difcult for them to focus or feel engaged within the course, or students also wrote about the challenge of Time management, having to balance many demands and time constraints from their various roles as students, workers, family members, and humans in general. As a result, Time management codes co-occurred with many other codes, since students would often identify time as the challenge, then go on to describe the diferent facets of their lives that made time management difcult.

S-C-16: "I work about 30 hours a week as well as taking 17 credits this quarter, so time will be a challenge. "

Balancing all these demands could be difcult, especially when students struggled to set their own routines or to maintain a semblance of work/life balance. S-A-60: "I only have so much time in the day/week for this class, other classes, and personal projects. It's honestly really rough to balance productivity and sanity :/"

Both Motivation and Time management challenges had to do with students' current self-regulation capacities. Taken in context with many students reporting burnout from sources like the ongoing pandemic, competitive academic environments, and other stressors outside the class, it is perhaps not entirely surprising that students reported these kinds of challenges getting in the way of their learning.

5.2.1

Feedback was deemed important, even though purpose of StudentAmp was unclear. While interviewed students found feedback to be important to share, they expressed uncertainty about the purpose of StudentAmp. When we asked students what they thought the tool's intended purpose was, all fve interviewed students expressed some uncertainty with two framing StudentAmp as a tool to improve the course. Students also compared StudentAmp to other feedback tools, such as direct emails, mid-term feedback, and end-of-term feedback. One student felt StudentAmp focused on and enabled conversation about a diferent context than other feedback they may have given: S-D-57: "The feedback I said on StudentAmp was more 'What's going on with you? What are your challenges?' Whereas the feedback I gave for my instructor and my TA was 'Is the way they're teaching us helping? Are they getting back to us in time with questions?' I think they were just two diferent types of context in terms of what was being asked from us as students. "

Two participants noted how StudentAmp was unidirectional: Professors could get feedback from students, but not the other way around. One participant didn't expect much beneft from StudentAmp because of this lack of bidirectional feedback: S-C-88: "[StudentAmp] is a place where professors could hear the voices of students, to some extent. But students could not hear what professor thought [...] in this case, it's not [a] participatory process, not exactly like that. " 5.2.2 Challenges beyond the scope of the class were worth sharing, but privacy matered. When discussing value and risks of sharing challenges with StudentAmp, interviewed students talked about how difering goals and relationships with instructors afected what challenges they shared.

StudentAmp asked students to share the biggest challenge in their life, where that challenge could go beyond the scope of the class itself. Three interviewed students noted how their biggest challenge was often beyond the scope of the course, including S-D-57, who provided a metaphor of school as one of many "bubbles" in life: S-D-57: "I've always thought that it's important that teachers or professors or people you interact with know a little bit about who you are and a little bit about what's in your surrounding bubbles. School is only one bubble of a student's life, so knowing all knowing a little bit about those other aspects about student life, you know family, emotional, work, relationships, friendships. Just knowing a little bit about those things can give you general knowledge of how it could be impacting the school bubble. "

Another interviewed student also recognized that life outside of school afects class experiences, but they had concerns with that overlap occurring: S-A-148: "Stuf in your personal life defnitely afects class, but because this is a school thing it makes me not want to 'cross those wires' almost. I'm worried about inappropriate timing, or something. I don't want to be 'that person.' Which is weird or doesn't really make sense, because that's what the survey is asking about. But I don't know. That can kind of be the 'little fear' in the back of your head. "

Students perceived multiple risks to privacy and safety that sometimes limited what they shared through StudentAmp. One student noted that they didn't want a person they lived with potentially seeing what they wrote while they interacted with the tool, and therefore opted not to share particular challenges. Another student noted how professors may not be the best person to respond to certain types of challenges, such as mental health: S-A-128: "Maybe I could share these mental health [challenges] but, I don't know. I think I wouldn't just share those with a teacher, because you could share them with a professional who would be able to better help. "

Sharing about themselves made students feel vulnerable even when supposedly anonymous. One student justifed this risk by feeling it was necessary as part of validating their challenges to the teaching team: S-A-148: "I just hope that a professor would never think that I'm trying to take advantage of their kindness. [...] I wanted to let [my professor] know that I was serious, while staying anonymous, while asking for an extension specifc for me. So I don't really know how you can satisfy all that. " 5.2.3 Demographic information was seen as an asset, although risk of re-identification existed. Interviewed students identifed ways that sharing demographic information could help instructors understand the positionality of students facing diferent challenges. However, they also identifed potential risks related to acceptance of minoritized identities and potential re-identifcation.

The students we interviewed generally felt that demographics could help instructors interpret challenges that students shared, with a few caveats. For instance, one student felt that instructors should have the cultural competence ( [86]) to understand how societal structures afect learning experiences of students from minoritized groups: S-D-57: " So if an as an instructor sees that the students who are BIPOC [...] and they're not doing as well as white students. I feel like a good informed instructor would know the racial understandings and the gender understandings as why certain groups with demographics will not be doing as great as other [groups]. Simply because of the world we live in, and the kind of.. structure our society is built upon. So I think a good instructor would know how to interpret that information and how to better help those students because they're all just trying to be at the same end goal. "

Another student found it relevant that StudentAmp asked demographic questions relating to other life commitments. Multiple interviewed students were searching for jobs while also taking this course, with one saying how StudentAmp helped connect job searching to course experience in a way that instructors previously had not:

S-C-88: " I appreciate that [StudentAmp] cares about whether we have jobs. Because I previously chatted with some other instructors and they said 'for job searches, that's kind of something diferent. I frst care more about whether you learn well in this class. ' Which makes me feel like I need to separate the job searching and course work. But they are not separate things. They're defnitely things happening at the same time in my life. "

Interviewed students also shared concerns related to their identities being seen as valid by teaching team members. One student shared uncertainties about how accepting the teaching team would be with regards to their minoritized gender identity: S-A-148: "A risk might be [professors and TAs] don't take me seriously if they disagree with my identity or don't think my identity is valid [...] If this is a class I'm trying to do well [in] and take seriously, especially a class that's relevant to my major where I might see this professor again, or it matters a lot that I do well in this class, I'd be worried about not being taken seriously. "

Another student identifed the potential risk of re-identifcation. Even though courses were remote, teaching teams had additional information about students through their interactions with them as well as learning management systems. This information included a list of full names of all enrolled students. One student saw a potential risk of re-identifcation by connecting multiple pieces of demographic information with popularity of names in diferent cultures: S-A-128: "With more [demographic information], like frst-gen BIPOC, I feel like it would really narrow it down to a select few people [...] and there's tons of people who have similar names from certain regions. Like sometimes I can fgure out where someone's from based of of their name. "

5.2.4

Seeing others' challenges fostered community, but students questioned disrupt scores. Several interviewed students noted how seeing other students' feedback helped them feel less alone. Recall that students saw random pairs of their classmates challenges through StudentAmp (as described in Section 3.3.1). One student found that seeing challenges similar to their own made them feel less alone. Another student found the variety of challenges their classmates reported reassuring to see, especially during remote learning: S-A-148: " It was nice to see that there's a variety [of challenges], that people are going through diferent things, or getting diferent things out of the class. But then when it's the same challenges as me, that's also reassuring, because then I am like 'okay I'm not the only one that's facing this right now, or having difculty with this part of the class. ' "

Challenges that our interviewed students reported tended to have negative disrupt scores in the instructor view, suggesting that classmates found their challenges less disruptive compared to other challenges. StudentAmp aggregated meta-feedback responses into disrupt scores. Disrupt scores were the net number of times a classmate selected a given challenge over another challenge. The fve interviewed students shared 13 challenges which had a median and mode disrupt score of -2, with the minimum being -6 and maximum being +2.

Students tended to question the aggregate disrupt score associated with challenges they reported, especially given how low they were. This did not bother some students, as they still personally felt their challenges were valid. However, other students recognized the impact that negative disrupt scores might have on instructors' awareness of needs of minoritized groups: S-C-88: "As a user, when I see minus score, I would feel negative feelings defnitely there was some judging behind it. And I understand probably people want to use this way to sort the results to help people browse information efciently. But minority, disadvantaged, underrepresented people, they don't have many members or great numbers in the whole community. But still, they need to have their voice. It's not necessary because they are minority people and they have emergent needs, so other people would [...] probably be experiencing diferent things so that's my concern. "

One possible explanation for the observed variety in disrupt scores is that students interpreted the meta-feedback prompt diferently than intended. To better understand how students perceived the request being made of them on the meta-feedback page, we asked interviewees to recall and refect on their perceptions of it. During our frst interviews, one student recalled interpreting the prompt as we intended (i.e., that they should choose which challenge would be more disruptive if they personally had it), two could not recall what they thought of the prompt, and the remaining two noted being surprised or confused by the prompt. Whereas the goal was to have students select the challenge they found more disruptive, one student interpreted the prompt as asking which challenge they also had and another as which challenge best represented the challenge they wrote.

Another potential explanation for low disrupt scores was that how students articulated their challenges afected how their classmates perceived them. One student thought that some students did not select their challenge because they used informal language (e.g., "whack" to describe a level of difculty) and was not as verbose as some other students had been.

How teaching teams organized challenges reported in StudentAmp.

To understand how teaching teams interpreted challenges that students reported in StudentAmp, we analyzed teaching teams' processes for creating and assigning labels to challenges. Professors and TAs could create custom labels to represent categories or groupings of challenges, and then assign them to challenges that they felt ft into those categories. Some teaching teams focused on challenges most proximal to the course, such as those that dealt with course structure and course content, because they felt those challenges were most actionable. A TA in course A (T-A-6) read through all 139 responses in course A's week 4 feedback and labeled seven as "feature request, "foot_5 which were challenges they felt were actionable.

T-A-6: "'feature request' is a [label] name. It's just kind of actionable feedback that might help us make the course slightly easier for everybody and so it's more focused on [Course A] directly and things we do that may negatively impact how people learn. "

Rather than focusing on challenges that related to the course, other teaching teams looked at challenges more holistically. P-D worked with someone with qualitative research experience (outside of the research team) to analyze the frst feedback session and developed the following labels and descriptions:

(1) Mental Health: Distinct from Stress below, a condition (such as Depression) that is experienced by the student (2) Stress + Time: A (temporary) feeling resulting from excessive demands on time and energy, lack of time to complete work, challenges of work life balance (3) Motivation: difculties with being proactive, staying motivated, struggling to keep up, resisting burnout (4) Learning Environment: issues with the physical space (e.g., noise, distractions), issues with internet connectivity, also isolation: feeling alone, lacking a sense of community (5) Group work: difculty collaborating with other students (6) New to Coding: expression of intimidation, frustration, difculty getting started, feeling "lost" While P-D took a more holistic, top-down approach, a TA in course C (T-C-2) took a more bottomup approach by looking at the frst three pages of responses (75) in the frst feedback session and creating labels that refected causes of challenges. After TAs in course C worked together to label all 222 responses in the frst feedback session, the three most commonly used labels were online setting (43), workload (35), and social & collaboration (32).

While some teaching teams focused on challenges that were more directly related to the course and more actionable (e.g. course structure), other teaching teams considered challenges even if they were beyond the control of the teaching team (e.g. lack of social interaction in remote learning).

How teaching teams considered demographics.

In general, teachings teams tended not to consider demographics unless we prompted them during the interviews or unless they had prior training related to cultural competence. For instance, a TA in Course D (T-D-5) was familiar with speaking about equity and privilege from coursework in public health, and T-C-2 and T-C-3 were both currently enrolled in a course on educational equity and diversity. However, when members of the teaching team did consider demographics, they connected challenges to rich personas of students that deviated from expectations of dominant groups.

When Course B was reviewing challenges, they focused for several minutes on a specifc challenge from their data: "I'm unsure of my ability to train my brain to think this way" (the student reporting this identifed as part of several minoritized groups, including having mental and physical disabilities, taking their frst programming course, and being a transfer student; disrupt score -2 = 3-5). When considering the challenge, a TA (T-B-1) who had the same gender identity as the student who wrote the challenge connected the challenge to their own experiences as a student and thought to remind students in lab section that "it's normal to struggle a little bit; it is challenging material and you're learning really fast. " When prompted about the students' demographics, P-B focused on the transfer student label to identify implicit assumptions in the course design: P-B: "a transfer student that makes me think of someone who's more likely than not probably coming in from a community college so may have the academic background, but doesn't necessarily know the way to navigate a four year institution efectively [...] physical disability minor [...] that could be someone who may be wearing a cast [...] And then severe mental disability could be any number of things as well, but defnitely that would be something that would interfere with student's schedule or their ability to focus or their self esteem and their confdence and actually passing the course and and completing the assignments. " P-B then went on to propose improvements such as clarifying how to use university email and access the course's learning management system, reassuring students that they could succeed in the course, and granting individual extensions on assignments. Showing demographics did not necessarily translate to understanding on what to do with the information. When considering demographics for a challenge we labeled as job and course structure ("I work 40 hours a week 8[A.M.] -6[P.M.] so it can make it challenging to connect with TAs who only ofer ofce hours during the middle of the day during the week... " ), P-E was uncertain how to consider the demographic information this student who identifed as part of several minoritized groups, including being a transfer and frst-generation student, working full-time, and job searching.

P-E: "In my head mentally, I still often see students as the standard undergrad 18 to 20 year old [...] living on campus or an apartment somewhere. The 'non-traditional' students as they're often frame are a diferent kind of aspect. I'm not sure what to do here now...in my head, this a challenge for everybody. "

P-E framed problems of students from minoritized groups as similar to students from dominant groups, but more severe. And while demographic information challenged the archetype of the "standard undergrad, " P-E was unclear how to use this new information. P-D and P-E reviewed feedback together. While P-E was unsure how to consider demographic information, P-D connected this challenge from a student from minoritized groups to systemic challenges at the university: P-D: " what it feels like to read something like this is it is somewhere between heartbreaking and frustrating and angering. That is instructors were put in this really awful position where the university pressures people to take more courses than they can handle because [tuition] is so expensive. "

5.3.3 How teaching staf considered disrupt score. Disrupt scores were not taken literally, as the afordances of the interface design resulted in confusion amongst teaching teams and comparisons of such diverse challenges potentially confounding the aggregate disrupt score. Because StudentAmp ordered challenges by disrupt score, afecting how teaching teams viewed and interacted with the data. StudentAmp ranked in each feedback session by disrupt score, resulting in teaching teams seeing challenges ordered from highest disrupt score to lowest. Disrupt score was shown as a number that is the diference between a positive number next to a thumbs up icon and negative number next to a thumbs down icon, as shown in Fig. 2e. It represented the net number of times a students selected that challenge over a random other one when asked to determine which challenge they found more disruptive.

One TA looked at the frst three of nine pages (75 responses) and created and added labels to them, using the disrupt score as a stopping criteria:

T-C-2: "I just [went] over all the responses that are frst three pages of responses and try to categorize and that basically settled all the tags [...] I think there is like a thumbs up, thumbs down. So I guess students get to like and dislike, or agree or disagree with certain statements. So up to page three, getting to a point [the disrupt score] is up one. So I think that's probably enough for telling what the students fnd most challenging. "

But later on, as they submitted a response as a student to explore the student view, the instructor questioned their interpretation of disrupt score as a measure of how many students related to a challenge:

T-C-2: "the net disruptive score [...] I don't like the minus sign. The frst time I read this, I thought it means 'people do not agree with this.' [...] And then later on, when I actually [submitted] a student response, I pretended [I] was the student. I review the process, and then I realized it's asking which one is more disruptive instead of which one resonates more with your circumstance. I felt that there's a diference there and it's not really clear when I frst viewed it. "

The thumbs up and down icons were similar with iconography used in many software interfaces to indicate rankings, but indicated something slightly diferent because selecting a response over another is not exactly the same as "upvoting, " and choosing not to select a response is not exactly the same as "downvoting. "

The professor for course A felt that the disrupt score was difcult to interpret in part because the diverse content of challenges made some comparisons uninformative. They gave an example of the disruptiveness of the global pandemic as being far greater than anything related to this class: P-A: "The pandemic's huge, and to say something is less disruptive than a global pandemic that's not a very high bar. EVERYTHING should be less disruptive than a global pandemic. But it is nice that nobody is saying 'yeah there's something so wrong with this class, that is the biggest problem, even though there is a pandemic, ' I suppose that's a win. "

Teaching teams really began to question low disrupt scores which corresponded to challenges they thought were disruptive. Of the responses which reported a challenge, responses with the lowest disrupt scores in each feedback session varied in content, but tended to be either vague or involved challenges that were not relatable to most students. The challenge with the lowest disrupt score in our dataset (-19=1-20) related to a phone being a distraction: S-D-69: "My phone is the biggest challenge I am very addicted and it takes all of my focus during class times. "

TAs from course D labeled this challenge as Motivation. When reviewing this challenge, P-D wondered if other students were either downplaying the severity of the challenge or questioning its authenticity: P-D: "Maybe people sort of downplaying the severity of that [challenge] or maybe that addiction is maybe something some people don't think is real. "

After hearing P-D's comment, P-E proposed an explanation wondering how the "non-clinical" language and focus on the phone may have caused peers to not take this challenge seriously: P-E: "I wonder how many people are like 'oh haha yeah my phone is a joke too oh yeah no totally animal crossing is like defnitely like my biggest distraction at the moment.' Whereas it's supposed to being like 'I have severe problems focusing on anything and I'm constantly spending time doom scrolling,' like there's actual things are going on there, but because of how [challenges] are framed and presented, they get read in very diferent ways. "

The disrupt score deviating from expectations caused P-D and P-E to propose alternative explanations for disrupt scores that deviated from their expectations. These explanations included students misunderstanding the meta-feedback prompts or not taking challenges seriously because of the way students reported challenges or because peers were unfamiliar with the challenges P-A also noted a similar case where a challenge may have gotten a low disrupt score because it only afected a subset of students. For week 7 feedback in course A, we coded the challenge with the lowest disrupt score (-15 = 1-16) as location: S-A-117: "Time diference" After seeing multiple challenges mentioning time zone diferences with negative disrupt score, P-A acknowledged the impact of this challenge on a select few students: P-A: "There's a few [responses] on timezone diferences, but they are pretty consistently downvoted. Which I've heard enough now to believe that the timezone diference is a big deal for a small population of students. "

This theme of a low disrupt score for mental health related challenges also appeared in Course D. In week 1, a student who reported being part of several minoritized groups, including having a mental disorder which severely impacted their learning experience, stated a challenge related to severe depression and suicidal thoughts as part of week 1 feedback for Course D. This challenge received a disrupt score of -2 (6 -8). In response, P-D talked about mental health in their next lecture and provided links to university resources to support students' mental health. P-D: "So the students ranked severe depression and suicidal thoughts they rank that lower than the other thing you know eight out of 14 times which either means that students misunderstood the prompt or they misunderstand severe depression and suicidal thoughts. "

For week 7 feedback in Course A, S-A-202, who reported belonging to multiple minoritized groups including having a minor mental or social disorder which hindered their learning experience, stated a challenge related to depression and having to go to regular doctors appointments to manage their depression. This challenge had a disrupt score of 1, with 9 students selecting it over another random challenge and 8 students selecting a random challenge over this one. Upon seeing the lower disrupt score, P-A questioned the disrupt score and gave an explanation related to a lack of familiarity with depression: P-A: "'My depression, ' that's unfortunate. That should have a much disruption rating...I would bet you that the eight people who downvoted that don't have any history of mental illness or depression in themselves or their families, because if you know what that's likethat should be much higher. "

StudentAmp avoids reducing students to labels, supports perspective Taking

Another interpretation is that StudentAmp is harmful through its reductions of people's diverse lived experiences. Data technologies can promote material, symbolic, and other violences by reducing people to broad demographic groups and promoting incremental changes that do not address larger systemic issues [39,40]. To avoid data violence related to stereotyping, we designed StudentAmp to show multiple dimensions of demographic information, enabling consideration of intersectional identities for perspective taking. StudentAmp enabled aggregation and reduction of challenges (through user-defned labels), but not by demographic groups. These design decisions required teaching teams to consider challenges as contextualized by multiple dimensions of student demographics. We found that discussing demographic information was challenging for teaching teams, with members with more cultural competence (e.g. training in culturally responsive teaching) more able to lead these discussions.

Showing high-dimensional demographic information likely enabled consideration of intersectional identities in perspective taking. Rather than see one or two demographic labels (e.g. gender and ethnicity), teaching teams could see up to nine (enumerated in section 3.3.1). Making visible multiple demographic features at once provided a mechanism to discourage simple and harmful stereotyping and instead supported more nuanced perspective taking. Furthermore, this feature provided teaching teams the option to talk about aspects of demographics they were most comfortable with. For example, we noticed in multiple interviews how teaching teams discussed that a student was a transfer or frst-generation and how this was more common amongst BIPOC students. All fve professors and many TAs identifed as coming from dominant groups, so speaking about some aspects of demographics may have been less comfortable with (e.g. gender, ethnicity) and other aspects of demographics may have been more comfortable with (e.g. transfer student, frst-generation student). From our data, it is unclear what the impact of using more comfortable demographic labels in discussions about equity might be, since it could be an inroad to starting "more difcult" discussions, or it could obscure more direct systemic issues about gender and ethnicity. This could be an exploration for future work.

Organizing information in a scalable, equitable, and privacy-preserving way is an open design space

Yet another interpretation is that StudentAmp was not equitable because it did not draw attention to the needs of minoritized groups. We tried to show the information about challenges in an equitable way by doing two things: First, StudentAmp attempted to call attention to minoritized groups' needs by showing demographic information with challenges only if that person was from a minoritized group. This enabled perspective-taking behavior, as discussed in Section 6.3. Second, StudentAmp also ranked challenges by disrupt score in an attempt to organize challenges by disruptiveness, but teaching teams found disrupt scores to be a confounded measure. The goal of the disrupt score was to introduce a mechanism that organized challenges by disruptiveness and not frequency or commonness, as minoritized groups could make up a small proportion of students and/or have unique challenges. In our interviews with teaching teams, we found multiple instances of them being uncertain of how to interpret the apparent contradiction of low disrupt scores for seemingly severe challenges, such as mental health concerns. Potential explanations include StudentAmp providing an unclear prompt or explanation for the meta-feedback task, as evidenced by multiple interviewed students interpreting meta-feedback prompts in diferent ways and being uncertain about what they were doing. Asking a student to consider the disruptiveness of challenges that they may never have encountered for the purposes of calculating the disrupt score is also a difcult task. Future work can explore potential improvements including the efect of diferent challenge selection procedures (e.g. asking students to consider challenges that are similar to their own, or from people from similar demographic groups) or contextualizing meta-feedback with demographic information or other information to enable more informed consideration.

An improved disrupt score is only one of many potential mechanisms to support the organization of information in an equitable yet efcient way while also ensuring privacy and well-being of minoritized groups especially. A large remote course with hundreds of students is a dynamic environment where students' needs must be met in a timely manner. Efciency in data collection and analysis is key to taking action. Furthermore, students of minoritized groups may face challenges that are unique and unlike challenges their peers have, so there must be a way to organize information so perspectives of minoritized groups are not lost. Lastly, addressing students' needs often involves vulnerability and asking students to share information about their experiences both in and out of the class. Because students of minoritized groups are perhaps most vulnerable to privacy violations, we must ensure the privacy and well-being of minoritized groups in particular. These tensions present a rich design space for future work that values equity and human well-being while also wrangling with the pragmatics involved with the need to scale.

Review Board (IRB) as STUDY00008871. Supplemental material can be found at https://github.com/ codeandcognition/archive-2022cscw-xie.