Search for: All records

This paper presents the design and evaluation of an automated writing evaluation system that integrates natural language processing (NLP) and user interface design to support students in an important writing skill, namely, self-monitored revising. Results from a classroom deployment suggest that NLP can accurately analyze where and what kind of revisions students make across paper drafts, that students engage in self-monitored revising, and that the interfaces for visualizing the NLP results are perceived by students to be useful. 

This paper presents the design and evaluation of an automated writing evaluation system that integrates natural language processing (NLP) and user interface design to support students in an important writing skill, namely, self-monitored revising. Results from a classroom deployment suggest that NLP can accurately analyze where and what kind of revisions students make across paper drafts, that students engage in self-monitored revising, and that the interfaces for visualizing the NLP results are perceived by students to be useful. 

Writing and revision are abstract skills that can be challenging to teach to students. Automatic essay revision assistants offer to help in this area because they compare two drafts of a student's essay and analyze the revisions performed. For these assistants to be useful, they need to provide useful information such as whether the revisions are likely to lead to an improvement in the student's grade. It is necessary to better understand the connection between revisions and grade change so that this information could be displayed in an assistant. So, this work explores the relationship between the tf-idf cosine similarity of two essay drafts and resulting essay grade change. Prior work has demonstrated that identifying the revisions between drafts, then labeling each revision with the purpose behind why the revision was performed is useful to predicting grade change. However, this process is expensive because this sort of annotation is time-consuming for humans. Moreover, classifiers achieve lower accuracy than humans when predicting purposes. Using similarity measures instead of or as supplement to revision purposes may correct these issues, as similarity can be computed automatically and without the issue of classification accuracy. As such, the correlations between grade change and the similarity measure are compared to the correlations between grade change and revision purposes with the potential use-case of an automatic writing assistant in mind. Findings suggest tf-idf cosine similarity captures overall essay and overall grade change while revision purposes capture lighter changes that fix errors or cause the essay to read better. 

We present the design and evaluation of a web-based intelligent writing assistant that helps students recognize their revisions of argumentative essays. To understand how our revision assistant can best support students, we have implemented four versions of our system with differences in the unit span (sentence versus sub-sentence) of revision analysis and the level of feedback provided (none, binary, or detailed revision purpose categorization). We first discuss the design decisions behind relevant components of the system, then analyze the efficacy of the different versions through a Wizard of Oz study with university students. Our results show that while a simple interface with no revision feedback is easier to use, an interface that provides a detailed categorization of sentence-level revisions is the most helpful based on user survey data, as well as the most effective based on improvement in writing outcomes. 

Data augmentation has been shown to be effective in providing more training data for machine learning and resulting in more robust classifiers. However, for some problems, there may be multiple augmentation heuristics, and the choices of which one to use may significantly impact the success of the training. In this work, we propose a metric for evaluating augmentation heuristics; specifically, we quantify the extent to which an example is “hard to distinguish” by considering the difference between the distribution of the augmented samples of different classes. Experimenting with multiple heuristics in two prediction tasks (positive/negative sentiment and verbosity/conciseness) validates our claims by revealing the connection between the distribution difference of different classes and the classification accuracy. 

Pleonasms are words that are redundant. To aid the development of systems that detect pleonasms in text, we introduce an annotated corpus of semantic pleonasms. We validate the integrity of the corpus with inter-annotator agreement analyses. We also compare it against alternative resources in terms of their effects on several automatic redundancy detection methods. 

This paper is about detecting incorrect arcs in a dependency parse for sentences that contain grammar mistakes. Pruning these arcs results in well-formed parse fragments that can still be useful for downstream applications. We propose two automatic methods that jointly parse the ungrammatical sentence and prune the incorrect arcs: a parser retrained on a parallel corpus of ungrammatical sentences with their corrections, and a sequence-to sequence method. Experimental results show that the proposed strategies are promising for detecting incorrect syntactic dependencies as well as incorrect semantic dependencies.