the subsequent talks. Michael Leitch will then discuss the power and limitations of multiple levels of analysis by reporting on a study in which he coordinated in-the-moment analysis of video with insights from coding and creating profiles of understanding from large-scale conceptual surveys with open-ended responses. He further discusses the corroboration of his results with recent fMRI data. The last two papers in the symposium take up the issue of coordinating insights from in-the-moment video analyses with analyses generated through the use of new technologies. Chandra Orrill will present her work using Epistemic Network Analysis together with Knowledge in Pieces to shed light on patterns in teacher reasoning about proportional reasoning. Finally, Bruce Sherin will present his analyses of clinical interviews transcripts using computational methods for thematic analysis and generation of knowledge elements. His talk will take on the benefits and drawbacks of augmenting traditional human-generated analyses with computational models, including the importance of having a means by which we can make explicit links between KiP models and data. The symposium is arranged to be 60 minutes, with each presenter having an opportunity to share their paper, emphasizing the methods in the studies they report, followed by remarks by our discussant, Andy diSessa, who will focus on the methodological insights gleaned across the discussion of the reported studies.

This talk addresses a fundamental question posed by newcomers to the KiP community: How might I approach theory building in the KiP regime? To address this question, I introduce a framework for characterizing scientific theory-building processes proposed by Collins and Ferguson (1993), called epistemic forms and games (EFG). This framework casts the knowledge-building work of scientists as epistemic games, the play of which are constrained by epistemic forms. Epistemic forms are general templates underlying specific epistemic artifacts. The construction of particular artifacts is achieved by filling out the template slots. Forms often correspond with particular research questions. For example, a scientist wishing to answer the question: "What is the nature of the Bengal tiger?" might produce an epistemic artifact that is a list of characteristics of the animal, including its habitat, diet, and daily activities. The form underlying this artifact is a list, and the game played by the scientist to fill out the list template is the list game. Moves of the list game include adding, subtracting, merging and splitting items.

I propose that the EFG framework might be intentionally taken up by scientists and scholars to use as a map for organizing their construction of new knowledge. The framework suggests a general-purpose approach, which I break into five steps: 1) determine the research question of interest, 2) determine the epistemic artifact that best answers that research question, 3) determine the epistemic form underlying that artifact, 4) determine the epistemic game and moves for filling out that form, and 5) play that epistemic game to fill out the associated epistemic form and create an epistemic artifact, which answers the research question. I draw on the EFG framework to guide my own theory-building work in KiP.

KiP is concerned with questions of knowledge and learning. While there are myriad questions a KiP researcher might seek to answer, many of them can be organized into the following four categories according to their underlying goal: 1) modeling the composition of the knowledge system, 2) modeling the structure of the knowledge system, 3) modeling the dynamics of knowledge in use, and 4) modeling processes of change in the knowledge system. I unpack each of these four foci of Knowledge Analysis below and illustrate each with an example from my own research.

A class of research questions within the KiP agenda are concerned with the composition of the knowledge system. These questions focus on the character of the knowledge elements used by novices or experts in reasoning about phenomena from different spheres of life, from physical (diSessa, 1993) to social (Philip, 2011). A specific question of this sort, which I have investigated is: "What is the nature of knowledge used by novices to engage in epistemic games?" (Swanson, 2022). To answer this question, I produced a list of elements of intuitive knowledge I called elements of meta-theoretic competence, which students demonstrated in building theoretical knowledge artifacts. In their 1993 paper, Collins and Ferguson explain that the list game, which is played to fill out the list form, consists of adding items, removing, merging, and splitting list items.

A class of research questions within the KiP agenda are concerned with the structure of the knowledge system. These questions focus on the organization of knowledge elements used in cognitive activities such as obtaining measurable information from the world (diSessa & Sherin, 1996) and mathematical problem-solving (Levin, 2018). In my own work, informed by the constructs of coordination classes and strategy systems in the aforementioned papers, I have proposed a structure that I call an epistemic system, to model student and scientist engagement in epistemic games (Swanson, 2023). This structure includes a perceptual component (perceptual strategies) for extracting information from the world, and a cognitive component (epistemic net) for transforming the extracted information into theoretical knowledge by organizing it according to an epistemic form. The epistemic net is further decomposed into orienting, operating, and explanatory elements, which interact synergistically during the enactment of an epistemic game. The form underlying this artifact is a system decomposition, which is organized as a network of elements. The general moves involved in decomposing a system into a network of elements include breaking a system into smaller pieces and determining the relationships between them.

A class of research questions within the KiP agenda are concerned with the dynamics of knowledge in use. These questions have focused on the activation of knowledge used in making sense of phenomena (Wittmann, 2002) and mathematical problem-solving (Wagner, 2018). A question I have addressed in my own work is: "How do students use their existing knowledge to construct an explanation for the warming of a cold liquid to room temperature?" (Swanson, 2023). To address this question, I produced a multi-causal model to show how the components of a student's epistemic system were activated in response to the question and available information, and in response to other elements. The general moves involved in producing a multi-causal model include identifying system components that are relevant to a particular event and determining causal interactions between them.

A class of research questions within the KiP agenda are concerned with conceptual change, or changes in the structure of the knowledge system. These questions focus on changes in the organization of knowledge elements used in understanding phenomena from different scientific domains including physics (Parnafes, 2007) and mathematics (Iszák, 2000). A question I have investigated is: "How did students build on their prior knowledge to construct scientific understanding of Newton's Law of warming?" (Swanson & Collins, 2018). To answer this question, I modeled the change in students' thinking over time as a temporal decomposition, showing how a more normative scientific understanding of the warming of a cold liquid to room temperature built on their intuitive knowledge through a piecemeal process over the duration of a class discussion. In doing this, I produced an artifact that divided a segment of student reasoning into a sequence of smaller episodes, which Collins and Ferguson call a temporal decomposition. The general moves involved in producing a temporal decomposition consist of dividing a sense-making event into smaller sense-making episodes and arranging them in temporal order.

The process I have presented is a general approach to theory building in KiP research. My intention is to make this kind of scientific activity more approachable for newcomers to the KiP community. Beyond that, the EFG framework may be a useful tool for more experienced researchers in discussing our work. Clarifying our methods is important for broadening participation in our research agenda and for sparking innovation. Finally, while the approach I presented is relatively straightforward, it should be understood that the details in the moves of each epistemic game are often quite complex and it is therefore advisable to refer to the methods sections of published work engaging in the kind of game one wishes to play, to find reference models for playing that game.

Vectors are essential to the language of physics, yet many students struggle with basic vector operations (Knight, 1995;Wutchana et al., 2015). Research on student learning of vectors often approaches the issue from the perspective of identifying mathematical errors and misconceptions (e.g., Appova et al., 2013). With the aim of surfacing underlying intuitions and conceptual resources, I conducted an exploratory study of learners' reasoning about the independence of vertical and horizontal components of motion, a phenomenon central to vector kinematics. The study used a mixed methods approach to collect survey responses and interview data. The knowledge in pieces (KiP) epistemological perspective framed the research questions, design of the survey instrument, and analysis of data.

A convenience sample of high school students (n=112) and undergraduates (n=49) were surveyed in their physics classrooms prior to explicit instruction on the independence of vertical and horizontal components of motion. The survey instrument presented two scenarios. The first scenario asked which would reach the ground first, a dropped bullet or a bullet fired horizontally from a rifle, assuming that the ground extended in a plane a far as the fired bullet travelled. The second scenario asked the same question about two balls rolling off a table, one rolling slightly faster than the other. Students were asked to explain their reasoning in short essay form. The contrast between a dramatic difference in the horizontal component of motion with a small difference was designed to elicit context-dependent conceptualizations.

The survey data was quantized for two analyses. First, responses were tabulated into the space of nine possible answer combinations to the two questions as to which object lands first. I refer to these nine possible answer combinations as profiles. For example, a common answer combination was that the dropped bullet lands first but the rolling balls land at the same time (Profile E). The frequency distribution of profiles was similar between the high school and undergraduate student populations (Figure 1). Differences in the proportions of the profiles across the two populations were not statistically significant ( (3, N = 126) = 158.04, p < .001).

Second, grounded coding cycles identified explanatory utterances that students referred to in explaining their answers. In vivo coding (Saldaña, 2016) saturated after distinguishing 33 distinct referents. I considered these codes as knowledge resources in a general sense, without venturing into the issue of ontology. Examples include, because it falls right away, and, because gravity is the same. Comparing frequency diagrams of these coded elements across the different profiles suggested different patterns in the knowledge resources students used to justify their answers. For example, Profile D students gave explanations involving gravity in correctly stating that the two balls land at the same time. However, in explaining why they believed the dropped bullet would land first, these same students consistently referenced path lengths and inflections. Conceptual coding (Saldaña, 2016) generated medium-scale and large-scale categories of responses. Even at these aggregated levels, students appeared to be thinking differently about the two scenarios. Figure 2 suggests that students generally thought about path length and inflections more frequently in the first scenario and less frequently in the second scenario.

The similarity of the profile distribution across two sizable populations provided evidence that the patterns in thinking suggested by quantizing the survey data constituted a statistically significant signal, rather than noise. However, coded survey data is layers removed from students' internal states. I collected process data

Undergraduate Students (Parnafes & diSessa, 2013) through semi-structured interviews with five of the undergraduate students to corroborate the survey results and to investigate causes for the apparently different thinking in the two different contexts. I conducted three, 30-minute interviews with each student, once before instruction on components of motion, once immediately after instruction, and once at the end of the semester. Prior to the first interview, students had completed their surveys and I had completed analysis of the quantized data at the aggregate level. This allowed a discussion of the interviewee's survey responses to serve as the initial semi-structured interview task. This also provided a conceptual framework in which to operate as the interviewer, as well as develop a design on the run (Morse, 2010) research question: namely, when it occurs, what causes shifts in reasoning about falling objects between the two contexts. The timing of the second interviews allowed for the exploration of conflicts between the students' intuitive knowledge and instructed content. The third interviews provided a kind of delayed post-test.

Interviews showed that students' inferences were affected by the perception (visual or imagined) of the additional velocity in the horizontal direction; inferences that were both consistent and resistant to instruction. Specifically, longer time was inferred from longer path length, which was inferred from higher velocity. This result is consistent with earlier research on the geometrization of dynamically defined motion (Saltiel, et al., 1980), and consistent with recent fMRI studies on time-to-collision tasks in which, compared to distance variations, variations in velocity produced stronger illusory perceptions of longer trajectory lengths in the visual cortex (Li, et al., 2015). Findings from these different levels of inquiry appear to converge. From a KiP perspective, a type of primitive resource seems to strongly moderate perceptions and inferences as physics students learn about projectile motion.

Along with members of my research team, I have been using Epistemic Network Analysis (ENA; Shaffer et al., 2009Shaffer et al., , 2016Shaffer et al., , 2017) ) to understand teachers' knowledge of the math content they teach. To this end, we have been successful in using ENA combined with traditional qualitative analysis to explore teachers' knowledge for proportional reasoning. ENA is a useful tool for characterizing the ways in which knowledge resources interact for participants. ENA allows us to code for understanding at the knowledge resource level. As an example, in our work on proportional reasoning (Weiland et al., 2020), we identified knowledge resources middle grade teachers used to solve a set of novel proportional tasks. Some of the knowledge resources attended to the structure of proportions (e.g., ratio as measure, covariation, ratio as a multiplicative comparison, and reasoning about composed units). We also identified knowledge resources that attended to ways of solving proportions (e.g., unit rate, equivalence, between unit think, and scaling). Using ENA, we have been able to better understand similarities and differences between teachers in ways that would otherwise be unknowable.

To illustrate how we use ENA, I focus on three experienced teachers from our participant pool (n=32). Greg worked in an urban private school and consistently demonstrated a depth of mathematical knowledge we rarely see in teachers. Matt taught 7 th grade in an urban public school. He was teaching most of his students for the second year as he had moved from 6 th to 7 th grade with them. Matt demonstrated good flexibility with knowledge resources, particularly those he emphasized in his classroom. Tori taught 6 th grade math and science at a suburban middle school. Tori's knowledge of proportional reasoning seemed to be typical of the teachers we interviewed. She had an array of knowledge resources, yet they were not as tightly coordinated as they could be.

All three teachers completed a think-aloud protocol using a Livescribe pen to capture their spoken and written work. They all also completed an in-person clinical interview that engaged them with paper-and-pencil items as well as items that relied on dynamic sketches to explore aspects of proportional reasoning. We used constant comparative analysis with a code set that used both pre-existing and emergent codes to evaluate each utterance of the interviews (e.g., Charmaz, 2014). An utterance was the response to a single mathematics question. Consistent with ENA system requirements, each utterance was coded using a binary system. Each code was in a spreadsheet column and each utterance was intentionally examined for each code. A 0 indicated that the utterance did not include evidence of that knowledge resource whereas a 1 indicated that the utterance did include evidence of the knowledge resource. (a) Greg (b) Matt (c) Tori

As shown in Figure 3, the codes we identified as ways of calculating answers yielded similarities in the ways the teachers used their knowledge resources. They all relied on Scaling Up and Down (bottom vertex) as one of their main approaches. For Greg, Scaling was often used with Between Measure Space Reasoning. For Matt and Tori, it was used with Unit Rate and with Between Measure Space Reasoning. It is interesting that Matt and Tori both have relatively heavy reliance on Unit Rate Reasoning with other knowledge resources, whereas Greg does not. He clearly has access to that resource, but does not rely on it the same way the other two do. (a) Greg (b) Matt (c) Tori

When we consider structural aspects of proportions used to reason about proportional relationships, we see considerable differences between the teachers (Figure 4). Greg relied on Covariation and Relative Thinking heavily, while never using Batches interpretations. In contrast, both Matt and Tori relied on Batches thinking. Further, Tori did not use Ratio as Measure or Ratio as a Multiplicative Comparison at all. Matt and Greg both seemed to be able to coordinate other knowledge resources with these two concepts.

If a practical goal of research on teacher knowledge is to improve teaching and learning, researchers need to know what teachers already know to maximize the potential of professional development. Work like that reported here highlights that teachers already have access to a number of knowledge resources, however, they may not coordinate them in coherent ways. This work can inform both a different definition of teacher learningone grounded in connection making instead of focusing on a body of knowledge -and changes in PD to focus on providing opportunities for connection making rather than adding to teachers' knowledge bases. This analysis also highlights the relative lack of attention to treating ratios as multiplicative comparisons, a particularly robust way of reasoning with proportions. Thus, this analysis can guide both an approach to professional development (focused on connection making) and elements of knowledge that may be particularly lacking across diverse teachers (e.g., ratio as multiplicative comparison).

The program of Knowledge Analysis poses unique methodological challenges. The goal of Knowledge Analysis (KA) is to identify mental representations that underlie some types of human reasoning (diSessa et al., 2015). In that regard, its aims are not different from those of other, cognitively oriented, work in the learning sciences. However, KA makes assumptions about the nature of human knowledge that, methodologically, make it very difficult to implement. First, it presumes that, in the regimes that it investigates, knowledge will often be nonpropositional in nature. This means that, as analysts, we may not possess ready-made language for describing this knowledge. Furthermore, many programs in KA aim to produce knowledge-in-pieces (KiP) models. In these models, knowledge consists of a large number of smallish elements, which are cued in a context-dependent manner. The methodological problems faced by these KiP models are particularly severe. If we adopt a KiP model, multiple non-propositional elements may be implicated in any moment of reasoning.

Thus, in these cases, the connections between data and theory can be complex. We cannot assume that we can look at moments of reasoning and see knowledge elements. This is not necessarily a problem; in fact, it is how science often works. We create theories or models, and then we look to see whether our observations are consistent with the predictions of the models. There is no need to presume that elements of the model are directly "see-able." However, in the case of KiP models, the links between theory and data are more difficult to operationalize than, say, models in the physical sciences.

A new collection of analytic tools -computational methods -are quickly gaining prominence in the learning sciences (Siemens, 2013). Some of these methods can be used to automate the pattern finding work that, in the past was always performed by human analysts. Furthermore, these methods can be employed to, in essence, operationalize the link between complex data and models. Thus, it is possible that we can use some of these methods to explicitly operationalize, at least in a crude way, the link between KiP models and the associated data.

For approximately the last decade I have been attempting to do just this; I have been attempting to use computational methods to discover, in transcript data, the sort of knowledge elements that comprise a KiP model. If successful, these efforts would have a number of benefits. One benefit would be the reduction of work by human analysts. However, there are, I believe, other more important benefits; namely, the creation of these computational models requires us to surface and make explicit the links between KiP models and data.

In my earliest work along these lines, I examined a corpus of interviews in which middle school students were asked to explain the causes of the earth's seasons (Sherin, 2013). I began with transcripts of these interviews. These interviews were then split into small segments, and the segments were clustered. These clusters were then assumed to correspond, roughly speaking, to the elements of a KiP model. That analysis was successful in that it produced elements that were interpretable and aligned with the work of human analysts. However, the application of the methods, and the interpretation of the results, was highly ad hoc. Many decisions were made solely because they were necessary to produce interpretable results. In more recent analyses, I have made use of a topic modeling algorithm called latent Dirichlet allocation (LDA) (Sherin, 2015). LDA is based on a generative model that can, I argued, be interpreted as a kind of simple cognitive model. This leads to analyses that are less ad hoc, and for which the interpretation is built into the design of the methods.

While the results of these analyses have been suggestive, they have been largely focused on this one corpus of interviews about the seasons. But this corpus has a number of features that make it unusual. Most notably, the subject matter was tightly focused, and we were able to conduct the interview in a manner such that it followed one of a number of predictable paths.

Furthermore, there are good reasons to worry that these methods won't work on less focused interviews. Methods such as LDA are very good at identifying what a text is about. They are good, for example, at distinguishing portions of a text about the seasons from portions about dinosaurs. LDA is likely to find these types of topics if they are there to be found. It thus might require narrowly tailored interviews for these methods to discern the sort of deeper structures that align with a KiP analysis.

In this talk, I will dive deeper into these issues. I will try to explicate where these analyses work (when they do), and why they work. To do this, I will make use of some additional data corpora. First, the seasons interviews were drawn from longer interviews that considered a larger set of subject matter, including questions about climate. I will show what happens when more of the interview is included in the analysis. Second, I will draw on analyses of interviews with middle school students about human biology. These interviews range over a much larger range of kinds of subject matter. The hope is that this exercise will illuminate the work of KA researchers and help us understand how KA analyses are methodologically operationalized. For example, one question I will address is the way the KA perspective must be built into the design of the interviews themselves. I will also seek to understand how the analysis "sees" knowledge elements by looking within and across portions of the data.