Education involves richly semiotic processes that are often inscribed for analysis or lost to the research process. Whether the indicators and information used to study what goes on in learning settings are proximal to the activities we see as consequential for learning-classroom activity, groupwork, engaging with digital tools, etc.-or distal activities including tests and surveys, our research is constituted in part by the records that are made of student work. Over the last fifty years, the tools used for recording details related to learning have advanced from audio tape to video recordings. Video has evolved from single general-purpose cameras to multiple and special equipment (Goldman, et al., 2014) all in search of better representations of what is occurring to enable learning. Today, these familiar approaches have been joined by digital platforms with new ways to collect information about student activity that these platforms often structure.
This paper shows some contemporary methods of creating transcripts and other digital traces of student behaviors. These records show some things with clarity. Other things are less clear. Together these new inscriptional technologies provide important capacity to understand student actions over large numbers of individuals and events. In the past, advances in education and social science research have prompted reflections on the nature of these inscription/transcription processes and their interrelationships with the epistemological/theoretical positions of the researchers. This paper adds to prior reflections with examples from research involving instruction mediated by a range of technologies designed to investigate activities of students working together and learning about collaboration across both conversational and textual practices. This paper provides some examples of new forms of collecting and representing student work in the important and challenging area of collaboration.
This paper's intellectual lineage begins in 1979 with Ochs' essay titled Transcription as Theory that provided a complementary perspective to technological optimism that recording equipment was dramatically reducing the filtering of researchers because of the benefits of a mechanically produced record in the form of film or magnetic tape that could then be transcribed and analyzed. Ochs focused on the transcription process and choices researchers and transcribers make to render the recorded event ready for analysis. Her work was drawn from studying language acquisition in small children. Her essay discussed issues of conversational turns and overlapping communication acts, gesture, and proxemics must all be inscribed through notation that then becomes a record with specific affordances to enable (and constrain) different analytic processes.
In 2000, Hall wrote Videorecording as Theory that expanded the discussion drawing on examples from educational research. Hall discussed issues of context and how contextual information is depicted through choices of camera placement, cropping, and editing. Hall looked at interactions not only between participants but between participants and technologies and how interview protocols shape the resulting video records. Hall further looked at the sharing of video records in datasets and how the use of those datasets may or may not align with the nature of the data collection and how sharing of video records may raise privacy and ethical concerns. This paper draws on a contemporary educational research project featuring a digital simulation embedded within a curriculum to teach undergraduate students about the collaborative nature of careers in science, technology, engineering, and math (STEM). Information architecture is a concept that has historically been visual where presentation choices impact comprehensibility (Wurman, 1995;Authors, 2006). This project includes several kinds of student interactions that have been recorded (transcribed) through automated digital methods. While automated transcription of audio recordings has become commonplace, the use of this technology combined with other digital data collection leads to four ways this paper expands Ochs' (1979) and Hall's (2000) work:
1. Computationally derived data transcription streams with visualizations (Figure 3) 2. Collaborative textual practices as students interact with shared documents (Figure 4) 3. Integration of multiple data streams, including conversational and textual (Figure 5) 4. Multimodal discourse from students interacting in a digital simulation (Figure 7) Each of these innovations will be discussed later in this paper in the context of a National Science Foundation (NSF) funded research project to learn how a digital simulation embedded in a curriculum can support students' acquisition of STEM career awareness and development of career intent. This project focuses on collaboration and group dynamics and the records of, and analyses drawn from the digital environment are used for both research and supporting students in their class work. It is also using the data collected to support both research and students in the class by providing analytic data in reflection activities for instructional uses.
The project used to provide examples for this conceptual paper uses a design-based research (DBR) approach (Barab & Squire, 2004) augmented with pre/posttests, embedded tasks based on Expansive Framing (Engle & Conant, 2002), and learning analytics (Authors, 2020) used for iterative program development (Authors, 2018). The project is occurring across two sites: one a diverse middle-Atlantic public university and Mountain state private university with much less diversity. This paper draws from a single semester with an undergraduate class of 96 students conducted in-person with optional videoconferencing option that was utilized by 20-30% of the students each week for a variety of reasons, including health concerns, students as commuters, and convenience.
The students course blended traditional didactic instruction and a narrative curriculum that was fully embedded within the didactic and used the normal instructional and grading structure of the course as illustrated in Figure 1. The narrative curriculum had that utilized professional documents to provide students with productive disciplinary engagement or PDE (Engle & Connont, 2002), aligned with Gee's notion of discursive identity (Gee, 2002) and using language to teach professional understandings (Duschl & Osborne, 2002;Lemke, 1990).
A central goal of this research is to teach students about collaboration in STEM fields through an active immersion in a simulation of professional activity. The narrative design is based on a role-based jigsaw pedagogy (Dunleavy, Dede, & Mitchell, 2009), a cooperative learning variation (Slavin, 1980). The narrative for the curriculum and the simulation featured four professional roles intended to depict STEM careers: Public Health (Science), Public Information (Technology), Public Works (Engineering) and Data Science (Math). Consistent with Jigsaw methods, the class was divided into groups based on these roles and the students were brought together in interdisciplinary teams for the simulation where each role had specific abilities.
The simulation provided students collaboration opportunities through a task structure that involved learning about incidents and then responding to those incidents. Collaboration is operationalized and analyzed in terms of communication and coordination: the ways students communicate verbally, through structured communication channels (ex: chat), and within documents. These interactions in both the early and later (simulation) parts of the narrative were structured in pedagogical patterns that provide multiple opportunities for assessing learning as shown in Figure 2. 1 The curriculum in both its document and its simulation portions uses a three-part pedagogical pattern where students were assigned some work to do with documents then meet and discuss the documents and produce an artifact for grading as illustrated in Figure 2. The work the students do in the first part of the narrative is conducted on shared Google documents (Google Docs). This pattern is repeated weekly with the students working in teams as they learn the narrative in preparation for the simulation. In the simulation phase of the narrative, playing the simulator replaces the Google Docs activity in the pattern in preparation for the meeting.
Part of the implementation of this project was altered because of pandemic safety protocols. What was intended to be table-top groupwork was replaced with students working in videoconferencing sessions and submitting records of that meeting (video and transcript) for grading/analysis. This pattern was typically conducted over several days with the assignment on a Thursday and the students able to complete tasks for group submission on a Sunday night. By submitting their artifact, video, and transcript the students provide the instructional team information to grade and provide feedback (as well as automatically identifying participation for grading).
While creating a moderately accurate transcript of audio is commonplace. These generated transcripts have specific properties and affordances. As shown in Figure 3, a transcript with timings of speech comes from the videoconferencing software (a). The verbal interpretation is not always accurate, but the timing is. This file can then be converted into a dataset (b) that can be directly used in visualization software 2 . While the visualization (c) does not show specific detail about what is being said at different points in time, the information about who is speaking when and how much of the conversation is being produced by which participants is largely accurate. The depiction can be used to ask questions about the collaborative processes of the individuals present.
The conversational gaps presented in the transcript files have generated new research questions for this project with various theories as to whether these gaps represent time the students are working on the collective artifact and whether there are any discourse patterns associated with the periods.
A part of the pedagogical pattern involves students collaborating over shared documents. Giving students shared textual assignments-slides, documents, technical products-is commonplace in many classroom settings and common in many professional STEM settings. However, what research literature exists around collaborative document practices has largely been focused on direct observations of students working synchronously (Lowry, Curtis & Lowry, 2004) or in editing on collaborative web platforms (Stoddart, et al., 2016). Little work has shown how students work on shared documents asynchronously over time. Figure 4 shows the path that comments take from their state as students experience them in the margins of the document to a tabular format to then a data display like the one used for transcripts but showing activity over days.
This visualization allows different team engagement patters to be readily seen and compared. The size of the squares is related to the number of actions (comment or reply) taken. Team B, for example works over the course of several days well in advance of the deadline. All members of Team B comment and two also reply to existing comments. Team C, however, does not begin working until the day the assignment is due. None of this team's members replies. The differences in team engagement and when and how deeply they engage is a familiar speculation for instructors. These data provide a lens on that activity that instructors can use to pinpoint when students did this work and which team members did not.
In several the conversational transcript visualizations like the one shown in Figure 3, gaps appear in the conversation when no students are speaking. The research team speculated these gaps might be when the team is working to produce a shared document (see Figure 2). The research team further speculated that when there is an asymmetry in conversational time (as shown in Figure 3) that there might be a counterbalancing activity in editing. In other words, as one student is leading the conversation then other students are working on the Figure 1 -Document edits combined with meeting conversations document. When the shared document is a Google Doc, it is possible to see when edits and are being made to the document. This information is limited to only knowing when edits were made and by whom rather than the details of the edits, including which material was added/changed by a person 3 . This visualization supported neither of the research conjectures to some of the analyzed meetings. Rather, Figure 5 shows how the same person who is making edits to the document is also the one who is speaking.
The second portion of the narrative uses a digital simulation within a similar pedagogical pattern that was used with professional documents in the first portion. Rather than having a commenting activity precedes the group discussion and development of a team artifact, the team plays a simulation. Data from the simulation is then used in asking the team to reflect on how they worked and their understanding of STEM collaboration and careers.
The digital simulation involves the students responding to a seasonal storm using an interface that depicts an emergency operations center (EOC) for a community where notifications of incidents that require a response are sent to one or more players. The EOC simulator presents each team has a different set of incidents and unique copy of the map--all teams are playing simultaneously different variations of the narrative. The players are not initially aware of the incidents that are occurring in their version. Each incident can be resolved by one or more roles for a move cost and resolution of incidents results in scoring. Players discover incidents by moving unit markers specific to each role into different locations. When a player moves a unit marker, they are notified of incidents that exist near their locations. Two roles, the Public Information Officer and Data Scientist have additional information sources. Players are notified of both incidents that match with their role and those that match with other roles leading to information sharing opportunities. Players have three ways they can communicate with their teammates inside the simulation:
1. A group chat that operates like other chat systems allowing direct text messaging within the team 2. Forwarding of notifications 3. Placement of indicators (a digital pushpin) that is a persistent reminder of the location of an incident.
The simulation produces an activity log that shows the kinds of things that team members do and when. Figure 7 shows two different simulation teams' activity records. They show how the simulation unfolds and the kinds of actions different roles are taking. The technology supporting this project produces 25 of these visualizations with about the same effort as to produce two. This enables analysis on a large scale, albeit with less attention to nuance and detail. Many important factors may be less visible than with more focused methods. Additional visualizations are actively being developed to support further analysis.
As with the prior visualizations, the game simulation representations show activity over time. With the first team in Figure 7, there is not much activity until the fourth member joins. The second team was able to begin much more quickly. There are then differences in how each team used the moves they are allowed. The first team spends more time moving on the map compared to the time they spend scoring. The second team is more efficient in being able to score more highly. These representations are shared with the students in their third reflection so they can reflect on who did what in their simulation.
An important part of this paper is the relationship between the depictions of human activity (formerly called transcripts) and research possibilities. The representations that have been presented are computationally created and benefit from the ability to combine digital data in different ways. These new ways of seeing student activity in these group settings is leading to new ways to structure the research and the teaching of the course the research is conducted in. This reciprocal relationship between technologies and practices is an important element that sociotechnical research focuses on (Bijker, 1987) where technologies shape and are shaped by practices.
One of the ways these visualizations is impacting the study and teaching is to allow insights into asymmetric team member participation in group activities. Instructors that assign groupwork, especially in some undergraduate programs, have become familiar with reports of how some students do less than others, sometimes called social loafing (Karau & Williams, 1993). as well as how some students dominate others. Understanding what happens withing the groups is a challenge for instructors. These visualizations, however, provide some clues. When looking at the conversation logs as shown in Figures 3 and5, the level of contribution is shown. While this is not of sufficient detail to conclude what students are doing, the visualizations provide important clues. When there is an assignment that involves commenting on documents to prepare for a group meeting, students who engage early and many times can be seen. When a student has a small role in a meeting, if that student did minimal or no work on the prior activity as Figure 4 shows then more inferences can be made about contribution. If, on the other hand, the representation of document comments shows all members were prepared for the meeting and one member dominates then a different set of questions can be asked.
In none of the examples presented in this paper is there the level of confidence of an underlying model of behavior to grade students or make research claims without additional information being gathered. Still, when working at the scale of 96 students and 25 teams these visualizations and the way they were created provides an ability to do this work efficiently at scale. These visualizations are helping inform a new research agenda that can use computational tools to further analyze the data streams and push the boundaries of large-scale instruction and support. The nature of the teaching assistant's role may also evolve because of these representations: shifting from being a grader to being an analyst and problem solver.
This paper began with a review of transcriptions of recorded human interactions that included speech as well as what have been called non-verbal communication such as gestures and proxemics. Ochs's (1979) Transcription as Theory explained how the process of transcribing entailed theoretical commitments that were passed downstream to the analysis. Hall's (2000) update twenty-one years later titled Videorecording as Theory expanded the discussion to look at the use of video recording and how the ways that recording equipment are placed and how the records are edited as well as the use of these datasets are important considerations. Hall also included interactions with digital technologies in his discussion in a way that invites this paper written 21 years later in a world reshaped by a pandemic where groupwork can be done using a platform that can automatically produce a transcript and recording and where there is no need for a videographer to record a meeting that is conducted online. This current world has greater capacity for information collection and sharing and where a computationally derived transcript can be automatically produced through checking a few boxes. It is a world where many of the interactions that students can participate in can be structured in ways that shapes the data that can be collected from them. Such is the case with this project where the digital tools can be used at a large scale of almost 100 students with very little beyond development of software achieve this.
The world of social science and education research today is greatly changed from what it was just a few years ago. The advances in creating research artifacts from recorded activity are emblematic of broad changes that have occurred and are occurring with computational tools. These computational tools and their data streams are interconnected in a web of technologies that can operate instantaneously at great scale. This enactment was for 96 students and prior to ICLS will be another enactment with over 120 students currently scheduled for Spring, 2022. The ability to collect information similar to that shown in this paper efficiently at scale is helping the project to reflect on the changes being manifest through these new technologies. This reflection suggests at this time three areas for consideration.
1. Reconsidering evidentiary classifications. The kinds of data and visualizations presented in this paper do not fit neatly into quantitative or qualitative paradigms. While interactional data found traditionally in transcripts would be more often associated with qualitative approaches, these records are computationally ready for a range of analyses and with the number of students and teams with data collected can provide statistical power for analyses that may be done. Indeed, the project is able to a case-study based on computational analysis. In addition, the data and visualizations shown here are not specific to the research dimensions of this project. They are used as well for grading and support of students. In order to determine which students participated, the transcripts from all teams are computationally quantified to show who spoke and for how much. 2. Sociotechnical structuring. As Hall (2000) indicated in discussing the way that focus group questions can structure what might appear to be naturalistic in edited video, the tools used to support the curriculum structure interactions in specific ways. The Google Docs provide two ways for comments to be made: a comment and a reply. They show them in margins and do not allow comments to be related in other ways. The video conferencing software (Zoom) places participants in certain places and the automatic transcriptions tends to see only one speaker at a time so that in many cases the transcript loses the overlapping speech common in conversational analysis (Schegloff, 2000). The research adapts to these constraints. 3. Automating surveillance. The techniques used to collect traces of activity in meetings and from documents could be considered a form of surveillance. The meeting transcripts are required to show who was at the meeting, which addresses a common challenge with many undergraduate students: group project absenteeism. With this ability to know specifically who participated and when has opened new opportunities for understanding student behaviors in ways researchers and instructors share. The technologies make this kind of surveillance easy and there are instructional situations where being able to pinpoint participation is helpful. However, it raises questions about ethics and privacy.
Researchers tend to work in comfortable patterns using techniques that have been successful before. This is especially true for emerging scholars who will use templates developed by those already established in the field. The kinds of approaches shown in this paper will likely become commonplace in the coming years. Researchers may be tempted to see in these new technologies a revolution in inferential possibilities. This paper like the ones it follows provide an optimistic, but cautionary perspective.