More Like this

1. Investigating “Who” in the Crowdsourcing of News Credibility

   Bhuiyan, Md Momen ; Zhang, Amy ; Sehat, Connie Moon ; Mitra, Tanushree ( March 2020 , Computational Journalism C+J)

   null (Ed.)

   Concerns about the spread of misinformation online via news articles have led to the development of many tools and processes involving human annotation of their credibility. However, much is still unknown about how different people judge news credibility or the quality or reliability of news credibility ratings from populations of varying expertise. In this work, we consider credibility ratings from two “crowd” populations: 1) students within journalism or media programs, and 2) crowd workers on UpWork, and compare them with the ratings of two sets of experts: journalists and climate scientists, on a set of 50 climate-science articles. We find that both groups’ credibility ratings have higher correlation to journalism experts compared to the science experts, with 10-15 raters to achieve convergence. We also find that raters’ gender and political leaning impact their ratings. Among article genre of news/opinion/analysis and article source leaning of left/center/right, crowd ratings were more similar to experts respectively with opinion and strong left sources. 

   more » « less
2. Perceptions of Human and Machine-Generated Articles

   https://doi.org/10.1145/3428158  

   Tewari, Shubhra ; Zabounidis, Renos ; Kothari, Ammina ; Bailey, Reynold ; Alm, Cecilia Ovesdotter ( April 2021 , Digital Threats: Research and Practice)

   Automated journalism technology is transforming news production and changing how audiences perceive the news. As automated text-generation models advance, it is important to understand how readers perceive human-written and machine-generated content. This study used OpenAI’s GPT-2 text-generation model (May 2019 release) and articles from news organizations across the political spectrum to study participants’ reactions to human- and machine-generated articles. As participants read the articles, we collected their facial expression and galvanic skin response (GSR) data together with self-reported perceptions of article source and content credibility. We also asked participants to identify their political affinity and assess the articles’ political tone to gain insight into the relationship between political leaning and article perception. Our results indicate that the May 2019 release of OpenAI’s GPT-2 model generated articles that were misidentified as written by a human close to half the time, while human-written articles were identified correctly as written by a human about 70 percent of the time. 

   more » « less
3. Detecting Frames in News Headlines and Its Application to Analyzing News Framing Trends Surrounding U.S. Gun Violence

   https://doi.org/10.18653/v1/K19-1047  

   Liu, Siyi ; Guo, Lei ; Mays, Kate ; Betke, Margrit ; Wijaya, Derry Tanti ( January 2019 , Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL))

   null (Ed.)

   Different news articles about the same topic often offer a variety of perspectives: an article written about gun violence might emphasize gun control, while another might promote 2nd Amendment rights, and yet a third might focus on mental health issues. In communication research, these different perspectives are known as “frames”, which, when used in news media will influence the opinion of their readers in multiple ways. In this paper, we present a method for effectively detecting frames in news headlines. Our training and performance evaluation is based on a new dataset of news headlines related to the issue of gun violence in the United States. This Gun Violence Frame Corpus (GVFC) was curated and annotated by journalism and communication experts. Our proposed approach sets a new state-of-the-art performance for multiclass news frame detection, significantly outperforming a recent baseline by 35.9% absolute difference in accuracy. We apply our frame detection approach in a large scale study of 88k news headlines about the coverage of gun violence in the U.S. between 2016 and 2018. 

   more » « less
4. Climate news articles lack basic climate science

   https://doi.org/10.1088/2515-7620/ab37dd  

   Romps, David M. ; Retzinger, Jean P. ( August 2019 , Environmental Research Communications)

   Although climate change is arguably the most urgent issue of our time, the general public knows little about climate science. Here, we investigate how often five basic climate facts are conveyed inThe New York Timesnews articles covering climate change from 1980 to 2018. With only one exception, the frequencies with which these facts appear in news articles today are vanishingly small. This suggests that print journalism is a largely untapped resource for educating the public on basic climate facts.

    

   more » « less
5. Misinformation Detection Algorithms and Fairness across Political Ideologies: The Impact of Article Level Labeling

   https://doi.org/10.1145/3578503.3583617  

   Park, Jinkyung ; Ellezhuthil, Rahul Dev ; Isaac, Joseph ; Mergerson, Christoph ; Feldman, Lauren ; Singh, Vivek ( April 2023 , Proceedings of the 15th ACM Web Science Conference 2023)

   Multiple recent efforts have used large-scale data and computational models to automatically detect misinformation in online news articles. Given the potential impact of misinformation on democracy, many of these efforts have also used the political ideology of these articles to better model misinformation and study political bias in such algorithms. However, almost all such efforts have used source level labels for credibility and political alignment, thereby assigning the same credibility and political alignment label to all articles from the same source (e.g., the New York Times or Breitbart). Here, we report on the impact of journalistic best practices to label individual news articles for their credibility and political alignment. We found that while source level labels are decent proxies for political alignment labeling, they are very poor proxies-almost the same as flipping a coin-for credibility ratings. Next, we study the implications of such source level labeling on downstream processes such as the development of automated misinformation detection algorithms and political fairness audits therein. We find that the automated misinformation detection and fairness algorithms can be suitably revised to support their intended goals but might require different assumptions and methods than those which are appropriate using source level labeling. The results suggest caution in generalizing recent results on misinformation detection and political bias therein. On a positive note, this work shares a new dataset of journalistic quality individually labeled articles and an approach for misinformation detection and fairness audits. 

   more » « less