Online learning communities are well-supported by various theories of learning that emphasize the role of social interaction in knowledge building (Kožuh et al., 2015) and those suggesting that knowledge is constructed within a social milieu (Hemetsberger & Reinhardt, 2006). Many advantages are associated with learning in online communities including increased motivation, improved learning achievement, and perceived satisfaction (Dawson, 2006;Richardson et al., 2017;Zhao & Kuh, 2004). However, supporting and sustaining online learning communities is rather challenging because the asynchronous nature of online learning cannot guarantee in-time responses from peers or instructors. The lack of active interactions between participants makes them feel isolated and such feelings can lead to dropping out from the community (Lee & Choi, 2011;McInnerney & Roberts, 2004).
A body of literature has suggested the importance of interactions in online learning communities. Interaction is found positively related to learners' satisfaction and motivation in online learning environments (Dawson, 2006;Zhao & Kuh, 2004). Communication among learners is also highly valued by instructors due to their affordance to understand the class and activities (Stephens-Martinez et al., 2014). Many research has explored how teachers and/or forum moderators such as teaching assistants attended to the learning communities strategically to build relationships with students, facilitate communication and interaction among community members, and support the sustainability of the online communities (Swan, 2002;Tait, 2000;Vonderwell, 2003). However, many of these studies relyed on manual efforts and discussed the difficulty of providing timely support for online communities on a regular class (Brook & Oliver, 2003;Yukselturk, 2008), not to mention large-scale online communities with thousands of students.
Automatic support, on the other hand, has the potential to achieve aforementioned benefits while managing instructors' workload and improving learning experience. Various tools were developed to identify knowledge gap, recommend learning materials and provide instructors with reference to the class learning process (Goradia & Bugarcic, 2019;Kulkarni et al., 2020;Lakkaraju et al., 2015;Lui et al., 2007;Rafaeli et al., 2008Xing et al., 2019). Based on the students' participation behaviors in the online community, researchers developed prediction models using machine learning to identify at-risk students automatically so that proper help can be designed to improve their performance (Lakkaraju et al., 2015). Goradia and Bugarcic (2019) reviewed a number of recommender systems in health education and concluded that successfully identifying at-risk students could facilitate timely educational intervention. Aiming to facilitate instructor-student communication and track individual/community learning process, a few studies applied machine learning techniques to cluster and analyze online texts published by learners (Goggins & Xing, 2016;Lui et al., 2007;Trausan-Matu et al., 2014 ).
Another major area of automated online learning community support research is text generation. Text generation is the usage of computing techniques to automatically produce textual content that appears indistinguishable to human-written text. It can be used to generate reports, answer questions, collect information and prompt for feedback. Some task-based tutoring systems like CycleTalk, WrenchTalk used generated texts to automatically deliver support for learning tasks (Kumar & Rosé, 2011). Tang et al. (2016) also demonstrate the potential use of text generation in essay tutoring by providing automated next word suggestion. Given several words as "seeds" to start with, the language model will generate text accordingly. However, supporting online learning communities has not fully taken the advantage of the rapid development of deep learning on automatic text generation.
In this study, we explore the usage of state-of-art deep learning algorithms for text generation in providing automatic support for a massive online learning community, Scratch. Scratch is a large, online, text-based community for students to learn programming. We examined the power of Recurrent Neural Network (RNN) and GPT-2 for text generation using two million Scratch comments for providing automatic support in the online community. RNN-based language models are presumed to be state-of-the-art in text generation, demonstrating potential in providing writing support with automated word suggestions (Tang et al., 2016). GPT-2 is a powerful deep-learning-based language model released by Google OpenAI in 2019 (Radford et al., 2019). The overall goal of this research is to examine to what extent the deep learning-based text generation can offer efficient and meaningful textual support to the leaners in massive online communities.
Social support theory is the leading framework for providing support in online communities, as shown in Figure 1. Social support is defined as the service and aid exchanged by individuals Figure 1. Theoretical framework. (Cohen & Hoberman, 1983), emphasizing its interactive, helpful, and positive attributes. The positive relationship between social support and intended outcomes is commonly accepted in the research community (Hsu et al., 2018;Langford et al., 1997). Social support theory is conventionally applied in physical environments to explore the relation between individuals. However, as the digital tools and online community flourish, social support theory has been applied to much research in digital environments (Tsai et al., 2017). This study applies automatic text generation techniques to support online learning communities within the social support framework.
The importance of support has long been acknowledged by studies which examined the development of online learning communities. Lally and Barrett (1999) reported the development and analysis of an online learning community. They examined students' online interactions and argued that socio-emotional support is the important basis for academic work. Similarly, Charalambos et al. (2004) shared some interesting findings and valuable lessons in developing an online learning community, STAR-Online. When describing a successful online learning community, they mentioned the following characteristics: "There are capable moderators that provide facilitation, help, guidance and support as needed to the members of the community", and "There is mutual support among its members and sub-groups" (p. 139). The rest of this session focuses on emotional and community support and their influences on online learning communities.
Emotional Support The importance of emotions in online learning communities has long been studied in the past decades. Appropriate strategies to facilitate emotions are indispensable to success in online learning (Barbalet, 2002). In the community of inquiry (CoI) framework developed by Garrison et al. (1999), which has received significant attention and research efforts, emotional expression is identified as an essential component to be socially present in online communities. A body of studies focuses on students' motivation, anxiety, and confusion, among other achievement emotions related to performance in academic settings. In general, students' expression of different types of emotions often leads to different learning outcomes (Baumeister et al., 2007). In the context of large online learning communities such as Scratch, students' emotions such as happiness and anxiety are directly related to their engagement and persistence (Giannakos et al., 2014;Wilson & Moffat, 2010).
Emotional support, addressing emotions of members in online communities, has been found important to learners' learning and outcomes. Emotional support is described as affective assistance (Kahn, 1980), providing help with members in the form of empathy, caring, encouragement, and other sentimental reinforcement (Bambina, 2007;Klaw et al., 2000). It is assumed to be the most important factor to the participants' perceived social support (House, 1981). The importance of emotional support is also identified with online learning communities where it plays a critical role in student perceptions of social presence and collaborative learning (Cleveland-Innes & Campbell, 2012;So & Brush, 2008). Cleveland-Innes and Campbell (2012) examined the self-report data of 217 graduate students who attended online programs and concluded that socio-emotional support is essential online and presents in teaching and cognitive presence as well. Such support contributed significantly to perceived social presence and learning. Moreover, emotional support has been identified critical to student perceptions of collaborative learning, social presence, and satisfaction (So & Brush, 2008). Their work suggests that emotional support is necessary to reduce students' sense of distance and online learning environments shall be designed to provide socioaffective interaction among students. However, all aforementioned studies rely on interactions among students and instructors, while it remains unclear if such a task can be fulfilled through automatic support. By applying automatic text generation into online learning environments, this study aims to investigate to what extent automatic supporting can provide emotional support to online learners.
Community Support Another type of support in the social support framework is community support. Community support is to strengthen interaction and enhance a sense of community (Bambina, 2007), which describes a degree that one feels belonging in a group whose members share common characteristics and dependency with each other (Sarason, 1974). Community support is the key to the community sustainability (Bruckman, 1998). One goal in community development is to provide mutual support through social activities which lead to a sense of belongings and increased social bonds (Liu et al., 2014;Wills, 1991). In online learning communities, shared social activities may take the form of collaboration, information sharing, and interaction through discussion forums.
Through enhancing the sense of community, community support helps learners with their learning in online learning environments. A number of studies have identified a positive relationship between sense of community and students' engagement, motivation, and learning outcomes (Kang et al., 2007;McInnerney & Roberts, 2004;Phirangee & Malec, 2017;Rovai, 2002;Tinto, 1994;Xing & Gao, 2018). Specifically, students who have strong feelings of community are found more likely to persist in learning with better learning experiences and performance (Rovai, 2002;Tinto, 1994), those who lack sense of community are related with feelings of isolation and distraction, poor learning experiences, and increasing risks of dropping out (Lee & Choi, 2011;McInnerney & Roberts, 2004;Phirangee & Malec, 2017). With all the benefits of community support, facilitating communication among online community members is highly valued. There is a possibility that members would stop visiting the online community if they fail to receive responses or communicate effectively with others. If an online learning community is able to actively prompt communication among members and contribute to their sense of community, learners are expected to benefit from it.
With all being said, emotional support and community support are able to support online learning communities and are expected to help learners persist in learning communities, as well as improve learners' learning experience. However, most existing recommendations focus on prompting interactions among members (Rovai, 2002;So & Brush, 2008;Yang et al., 2017), centering manual work. Guided by social support theory, this study explores the ways of providing emotional and community support using deep-learning-based language models.
Much supporting online communities research has centered manual work. Studies indicate that instructors spend much time and effort in articulating course materials and designing activities that prompt instructor-student, material-student, and student-student interactions in online communities (Dawson, 2006;Rovai, 2002). In order to build relationships with students and develop intervention activities aiming at prompting further interactions, instructors and teaching assistants need to check on the platform regularly, monitor the community development and students' behaviors, and provide timely feedback (Dawson, 2006;Vonderwell, 2003). Moderators are also highly recommended to facilitate and support online learning communities (Charalambos et al., 2004;Lally & Barrett, 1999). Recent work suggests that hub members can also scaffold the online learning community, as they spontaneously follow instructors closely and connect many small learning communities (Brown et al., 2015). However, all these supporting strategies require a large amount of time and manual efforts.
Correspondingly, various automatic methods were developed to support online learning communities. Graf et al. (2009) proposed an automatic approach to identify students' learning styles to enhance instructor-student understanding and facilitate support in an online learning environment. Piech et al. (2015) employed machine learning to model students' online learning as they interacted with coursework. Some language-based agents are also designed to support online learning communities specifically. Kumar et al. (2007) proposed an online learning environment using a dialogue agent to support collaborative learning between students as well as to provide instructions. Ferschke et al. (2015) applied two language-based agents in an online learning environment to facilitate community member help seeking and to prompt celebrative reflection. Their study demonstrated the combination of communication contexts provided by agents could benefit students' engagement in discussion.
Deep learning is one of the most popular topics in computer science. As technology advances quickly, deep learning has achieved outstanding performance in many fields like pattern recognition, machine translation, and natural language processing. Deep neural networks and transformers are the most popular structures for implementing language models (Io & Lee, 2017;Vinyals et al., 2015). Language models implemented through these sophisticated algorithms are able to generate human-written like texts, hold human-machine conversation, parse and analyze input texts, make recommendations based on textual data analysis results, and so on. Deep learning language models have been widely used in many industries. Given sufficient data, the language model can generate various reports on games, sports, weather, natural disasters, or traffic (Garcelon et al., 2018;Gkatzia et al., 2016;Io & Lee, 2017). This study aims to apply cutting-edge language models to provide conversational support for online learning communities.
Scratch is a massive online programming learning community released in 2006. According to the platform, over 47 million registered users worldwide created 24 million studios leaving 221 million comments till November 2019. To benefit early programming, one of core design principles for Scratch is to be more social than other programming environments (Resnick et al., 2009). The concept of sharing is achieved by a series of learning activities. Based on the provided blocks or others' uploaded work, learners are able to create various projects like games, book reports, animation videos, music videos, greeting cards, and tutorials, etc. Besides viewing, downloading, and revising others' projects, users can leave comments or vote for a project as well. These activities generate a tremendous amount of data. This study employed a data set of 2 million comments collected from the Scratch online community as the training set (Hill & Monroy-Hernández, 2017). According to the website, the number of registered users on Scratch peaks at the age of 12. In comparison with random textual data, our data set will be able to generate conversation genuine to the Scratch learning context.
Figure 2 overviews the methodology of this study. First, collected data are preprocessed for training purposes (see 4.1). As a result, redundant data and features are removed. Second, we compare the model performance between an RNN language model and a GPT-2 Small model using partial preprocessed data (see 4.2 and 4.3). Automatic measurements are applied during this step (see 4.4). Third, we use the whole data set to train a GPT-2 Small model. Then, we apply both automatic and human evaluation (see 4.4) to assess the performance of our model.
Some textual contents are supposed to be removed as they are unnecessary to the goal and may compromise the model performance. (1). Non-English texts. Although Scratch has set separate forums of different languages for its users all over the world, users may still post in any forum using any language. This study focuses on English only and removes all other texts. (2). Uniform Resource Locators (URLs), or web addresses. Numerous comments are associated with one or many URLs to share or distribute users' projects/interests. However, URLs' content does not make sense to human conversation, whether it is legitimate or not. Therefore, URLs are removed. (3). Other invalid characters and texts, such as non-ASCII (American Standard Code for Information Interchange) characters and empty strings. Computationally, invalid values are unacceptable to the model. These are associated with the way how data are stored in original files and decoding input data to train the model. Without proper data preprocessing, issues (1) and ( 2) result in a model generating texts with non-English terms and/or seemingly URLs, while issue (3) leads to experiment errors.
Several libraries are commonly used to detect language in Python 2, they are TextBlob, langdetect and langid. With the provided Google Translate API, library TextBlob can be used for language translation and detection. To detect one text is considered as one visit. To filter out all non-English texts of our data set needs about 2 million visits. The API, however, denies frequent visits in concerns of resources abuse, which makes it unsuitable to our study. Library langdetect is also undesirable as its performance on short texts is non-deterministic. In other words, the same text may end up with a different detection result each time. In comparison with langdetect and other language detectors, langid achieves a higher accuracy across all domains, including microblogs (Lui & Baldwin, 2012). This paper employs langid to address issue (1). Other irrelevant texts are removed via regular expressions and ASCII indexes in Python 2.
A deep conventional neural network (CNN) typically has three layers: an input layer, a hidden layer with multiple layers, and an output layer. The sophisticated architecture in the hidden layer represents and learns from the data, tunes the parameters, or in other words, trains the model. As the multiple layers in the hidden layer always feed the data forward, CNN is also known as one of the feedforward neural networks. RNN is an adaption of a conventional feedforward neural network. As the name suggests, RNN has a recurrent hidden state in the hidden layer that integrates information from the current time and previous time. Figure 3 shows the structure of a standard RNN. The left-hand side is an overall structure and the right-hand side is the unfolded hidden layer. This structure enables RNN's capability for dealing with sequential data. The generative result of an RNN is a probability distribution over the next value following the given sequence.
For sequential data, the order of each input matters. The order of words is important to a sentence. If the order is changed, the meaning of the sentence may change greatly. The same rule applies to paragraphs and documents. Therefore, texts can be taken as sequential data in deep learning. RNN has been widely used to deal with sequential data, and various RNN-based language models has been developed accordingly. As a matter of fact, RNN-based language models are presumed to be state of art in many text generation fields (Vinyals et al., 2015).
GPT-2 is a transformer-based language model released by Google OpenAI in 2019. Transformer is an attention-mechanism-based architecture (Vaswani et al., 2017). It transforms one sequence into another one with the help of Encoder or Decoder without implying any Recurrent Networks. Encoder positionally encodes the sequence. Decoder decodes the positional sequence. Despite being extremely sophisticated, the goal of GPT-2 is rather clear: to predict the next following word to the previous one.
GPT-2 is trained on a data set of 40GB of Internet texts ending with 1.5 billion parameters. It has shown such outstanding performance on text generation that Google decides not to release the fully trained model in fear of malicious use such as fake news or e-mail composition. As a matter of fact, GPT-2 holds state of art results in 7 out of 8 testing languages (Radford et al., 2019). A small version with 124M parameters and a medium version of 345M parameters are released to the public for research and experiments. The small version has 12 stacking blocks and the medium version has 24 stacking blocks (Figure 4). In this study, we use a small data set of one month's comments to train a standard RNN language model and a GPT-2 Small, respectively. Using automatic measurement, we will select one language model and train it with the whole processed data set.
Natural Language Generation (NLG) evaluation methodologies are conventionally broken down into the following categories: intrinsic methods that involve subjects to read and rate the texts, like output quality and user rating; extrinsic methods to evaluate how the performance effects user or system task success (Belz & Reiter, 2006). These methods either employ automated measurements coherent to the model or involve human effort to evaluate the performance (Gkatzia & Mahamood, 2015;Hastie & Belz, 2014). This research adopts both automatic measurement and human evaluation. The model loss which is used to track the training process helps to determine whether the training is completed. The Flesch-Kincaid Grade Level is an automated assessment of output quality which measures the understandability of an English text. In addition, human evaluation is used to assess community and emotional support delivered in generated texts.
Flesch-Kincaid (F-K) Grade Level Flesch-Kincaid Grade Level was first designed to test the understandability of an English text for the US Navy (Kincaid et al., 1975) and later became a military standard. It is also widely used in research, for example, to access readability of medical or educational publications (Dufty & Graesser, 2006;Williamson & Martin, 2010). Unlike some other readability tests, F-K Grade Level calculation does not require a minimal text length. Sentence length and word length are the building blocks of F-K Formula's, which is described as: FK Grade Level = 0.39 * total words total sentences + 11.8 * total syllables total words -15.59
where the output is the approximate years of education or grade level needed to understand the text. This is implemented with the Python textstat library.
Human Evaluation In addition to F-K Grade Level, we also manually examine the automatic generated results to evaluate the magnitude of community and emotional support. Specifically, we compare the quality of support provided in the original replies with the support in the automatic replies generated by the deep learning language model. According to the power analysis results with a medium effect size, 79 comments with a reply are drawn randomly from the data set. Using each comment as a prompt, the trained model is able to generate an automatic reply accordingly. Then, coders rate the replies without knowing which reply is which. In the following of this study, we will use prompt, original reply, and generated results to describe the original comment, reply, and generated reply, respectively.
The studies of Lally and Barrett (1999) and Charalambos et al. (2004) have concluded a number of characteristics for successful online communities: mutual support among members; community-based facilitation, guidance, and support to members; and opportunities for socio-emotional discourse exchange. Adopting from their framework, several indicators are used for coders to examine the emotional support in a reply: (1) Whether there is an affective expression, including the use of acronyms such as "XD" and "LOL". (2) Whether the affective expression properly and positively addresses the current situation. An affective expression which indicates sarcasm shall not be considered supportive. (3) Whether the affective expression shows empathy, caring, encouragement, and other sentimental reinforcement with the member. Several other indicators are used for examining community support: (1) Whether the reply addresses the issues of a member. (2) Whether the reply is able to prompt interaction. (3) Whether the reply provides recognition, suggestions, and help, and increases the sense of belonging. Given a prompt, raters are asked to rate both the original reply and the generated reply without knowing which reply is which. Rating addresses the following questions:
. I found the reply provides emotional support.
. I found the reply provides community support.
The rating falls in a 1-3 scale, where 1 indicates Weak, 2 for Medium, and 3 for Strong. When coders can hardly or vaguely identify a type of support, the reply is thought to be weak in support. Medium support indicates that the type of support can be explicitly identified by meeting one or two indicators, yet support is not the theme of the reply. Strong support suggests one that can be explicitly identified and support is the theme. Examples are shown in Table 1. For each example, the first sentence is a prompt, the second one is a reply. The reply is either original or generated by the model. All texts are shown as what they are without grammatical correction.
Both models are run on an i7-9700 CPU, 3.00 GHz, 32.0GB RAM computer in Python 3. The RNN model is implemented using TensorFlow with eager execution. GPT-2 is imported from the gpt_2_simple module. The selection experiment is executed with one month of data. The average running time for each epoch of the standard RNN model is 1700 s, 30 s for the GPT-2 model. Figure 5 shows the running time of these two language generation models for 20 epochs. For visualization purpose, the vertical axis shows the natural logarithm to the actual running time. GPT-2 runs much faster than a standard RNN model. This study proceeds with the GPT-2 model.
We train a GPT-2 Small model using the entire preprocessed data set. After over 400 training steps, the average loss reaches its minimal value at 3.70 and continues to increase after that. Another parameter that affects the performance of generated results is the temperature, whose value controls how predictable the results are. In this study, we set it between 0.8 and 1.2 randomly to cover both predictable and surprising results. We randomly select 300 generated results and calculate their F-K -"no idea. xD" strong -"My stomach has been hurting reeally bad all day DX" -"hope it gets bettur!!" Community support weak -"anyone here?"
-"i might have to go soon … " medium -"im not sure when it will come out i will go check now wait."
-"29 of april i presume." strong -"Fang frowned. He had a little sister in the lab … . but she was dead now." -"I wonder why Im here, though. Is there anything I can do?"
Figure 5. The natural logarithm of running time in seconds.
Grade Level. Figure 6 shows the overall grade level distribution of the randomly generated results. In the figure, each dot represents one generated result. Most of the samples score between grade 0 and grade 10 and highly congregate under grade 5. This aligns with the user age distribution of the Scratch community, indicating readability of generated results fits the online learning community.
In addition to evaluate the readability of a text, F-K Grade Level has the ability to detect ungrammatical sentences and the outliers. In Figure 6, those few dots which lie away from the cluster can be seen as outliers. Table 2 lists generated results of different F-K Grade Level values. Generated results with a high F-K Grade Level value are indeed hard to understand. One speculation is that the model learns from the Internet expression practice which uses repeated letters or words to address one's opinion or feeling, such as "Soooooooo Annoying" (see Table 1). The usage of acronyms like "XD", "lol", and "btw" by generated results also indicates the model has adopted such an expression style. The average F-K Grade Level score of the randomly generated results is 4.02, with a standard deviation of 3.64. To summarize, most generated results congregate around grade level 5 with a small number of exceptions.
We further qualitatively evaluate the results in terms of generating emotional and community support. Two advanced doctoral students with extensive experience in computing and education Table 2. Examples of generated results with different F-K Grade Level scores.
Randomly Generated Results
Lol, basically, that means you were the person I used to call Ren and the dragon, who I go by much more than you, I make up I now call yourself the dragon that I call Ren and the dragon that I call Ren and the dragons that I call Ren and the dragons that I call Ren and the dragons that I call Ren and the dragons that I call Ren and the dragons that I call Ren and I believe "Yura was talking about her life, but she was thinking of everything else.// If I survive, I will be the leader of a clan that will protect the area" 7.2 "I think I was doing a bad job of my friends, because someone is NOT giving me a lot of stuff. XD" 4.8 AWESOME btw. im not that bad at math i do it on my laptop and i put it on 4.8 "The computer was running down all the rooms, trying to lock the door … " 3.7
"Okay, better look at the description. XD" independently rate the emotional and community support for original reply and generated results on a Weak, Medium, and Strong scale (see 4.4). Then, we calculate their inter-rater reliability for emotional support on original reply, and generated results, as well as community support on original reply, and generated results. The initial Krippendorff alpha coefficients are 0.71, 0.75, 0.70, and 0.64, respectively. After further discussion, raters reach 100% agreement on all ratings. Table 3 reports results of human evaluation. A supportive reply is defined as a reply whose rating is Medium or Strong. A reply is considered non-supportive if its rating is Weak. Two types of percentage of supportive replies are calculated. First, we calculate the percentage of supportive replies which provide emotional or community support. Supportive reply percentage on emotional (or community) support is the total number of supportive replies that provide emotional (or community) support over the sample size. Then, we calculate an overall supportive replies percentage which includes both emotional and community support. In other words, a reply is considered as overall supportive regardless the type of support it provides. The overall percentage is the total number of overall supportive replies over the sample size.
Table 3 indicates the language model is able to provide emotional and community support. Measured by percentage, generated results provide an equivalent amount of emotional support to the emotional support in original replies (p > .05). Meanwhile, both raters find that our deep-learning-based language model provides more community support than the original reply does. There is a statistically significant difference between the amount of community support in original reply and generated result (p < .05). Although the difference between overall supportive reply percentage in original reply and generated result is small, MANOVA test results indicate a statistically significant difference between the amount of support provided by original reply and generated result (p < .05). To summarize, the deep-learning-based language model is able to provide online learning communities with more emotional and community support than community members do.
Despite the various advantages of participating in online learning communities, supporting and facilitating online communities, especially massive online communities, are rather challenging (Brown et al., 2015;Swan, 2002;Tait, 2000). Due to their online asynchronous nature and scale, supporting online learning communities raises methodological questions for both researchers and practitioners as they strive to provide timely and quality support for community members. Traditional manual support methods used in small online classes (Rovai, 2002) may not meet the unique requirements posted by large-scale online communities. While there is some exploratory research on automatic support for online communities such as prediction mechanisms (Lakkaraju et al., 2015), and textual support using Quick Helper and Bazaar (Ferschke et al., 2015), these studies did not fully exploit the recent development of deep learning for building automatic support. This study proposed and examined the use of state-of-art deep learning language models to demonstrate their potential use in providing social support for a massive online learning community. The evaluation results show that deep-learning-based language models, especially the GPT-2 model, can provide stable and reliable support automatically to online learning community members. Such an automatic nature and reliable performance have significant implications for support research in online learning communities. Interaction is the principal way for online learners to perceive social support whose importance has been addressed by a body of literature (e.g. Hsu et al., 2018;Phirangee & Malec, 2017;Rafaeli et al., 2008;Swan, 2002). Meanwhile, prompting interactions among learners has long been recognized as an essential strategy to develop and maintain online learning communities (Charalambos et al., 2004;Dawson, 2006;Lally & Barrett, 1999;Rovai, 2002). However, many comments and questions are left unanswered or have a tremendous delay in response in the online learning community. Even in our own examined 79 examples, the response delay ranges from several minutes to several days or longer. When learners post to seek help or express confusion, delayed responses or the absence of response will not only impact the learning experience, but also potentially lead to the feeling of isolation and drop out (Lee & Choi, 2011;McInnerney & Roberts, 2004). By employing our deep-learning-based automatic textual support in a brutal force manner, these concerns can be relieved significantly.
Our design can also be integrated with other commonly used machine-learning-based techniques to support online learning communities. A number of studies develop classifiers to identify posts of interest, such as urgent posts that need to be addressed immediately, posts that express confusion or seek help (e.g. Almatrafi et al., 2018;Ferschke et al., 2015). Integrated with these classifiers, our design can be deployed to provide in-time support to those members in need. Such an "identify & support" integration leads to a fine framework which provides the online learning community with characteristic support automatically and systematically. Meanwhile, our design can be used to manage the instructor's work load. In an online teaching scenario, deep-learning-based text generation can automatically generate a number of replies at instructors' disposal. Generated responses can either serve as the "seed" for instructors to work with, or be used to reply a comment.
This study has methodological implications as well. Although various frameworks have been used to study support in online learning communities (Richardson et al., 2017;So & Brush, 2008;Zhao & Kuh, 2004) highlighting the importance of social support, sense of community, and emotions to online learning (Bambina, 2007;Cleveland-Innes & Campbell, 2012;Kožuh et al., 2015;So & Brush, 2008;), limited work has actually applied these frameworks to guide the design and development of automatic language support specifically. Guided by the social theory, this study demonstrates deep-learning-based text generation language models can be used to support online learning communities. With sufficient data, members of online learning communities can receive genuine responses like human-written texts. Our work has demonstrated that the automatic generated results using deep language model offer not only a grammatic response, but also uses the Internet writing style. While this study is focused on an online learning community, the methodological framework can definitely be applied in other settings as well.
There are some limitations to this work as well. First, the design of this application is in a one-toone manner: each comment gets one reply only. The evaluations on emotional and community support are based on such results as well. A richer context like continuous conversations may project different evaluation results. Second, each comment always gets a reply. In other words, each comment has a generated reply no matter what, even if it meant to close a conversation. Therefore, the application treats each comment equally, which may not suit the real world. Third, the readability of generated results cannot be guaranteed. As shown in Table 2, some of the results do not make sense. The randomness of such compromised results may impact users' experiences of a large group in actual use. This is only an exploratory work where GPT-2 is applied to support a large-scale online learning community for emotional and community support. Much can be done to extend or improve this work. Future work can explore how to integrate a text classifier with a text generation language model to address a specific type of posts. The quality of generated results can be improved by testing the readability of the generated results before displaying it. It is also worth to investigate how to deploy the design, for example, initiating a private conversation page that prompts interpersonal communication, or leaving a response in its original page. Future studies can also replicate the research findings in other online learning communities to examine their robustness and transferability.
This study took an initial step toward building deep-learning-based language models to provide automatic support to massive online learning communities. Specifically, this work employed the state-of-art deep learning algorithm GPT-2 in building such a language model and examined its effectiveness against the RNN algorithm. The results showed that the GPT-2 based text generation model not only outperformed the standard RNN model in execution time but also provided reliable emotional and community support to the learners. The findings have significant implications for large-scale support research and practice in online communities. By introducing deep-learningbased language models into the field, this study extends automatic support research in online learning communities.
H. DU ET AL.