Title: Photo sleuth: combining human expertise and face recognition to identify historical portraits
Identifying people in historical photographs is important for preserving material culture, correcting the historical record, and creating economic value, but it is also a complex and challenging task. In this paper, we focus on identifying portraits of soldiers who participated in the American Civil War (1861-65), the first widely-photographed conflict. Many thousands of these portraits survive, but only 10--20% are identified. We created Photo Sleuth, a web-based platform that combines crowdsourced human expertise and automated face recognition to support Civil War portrait identification. Our mixed-methods evaluation of Photo Sleuth one month after its public launch showed that it helped users successfully identify unknown portraits and provided a sustainable model for volunteer contribution. We also discuss implications for crowd-AI interaction and person identification pipelines. more »« less
Mohanty, Vikram; Thames, David; Mehta, Sneha; Luther, Kurt(
, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence Sister Conferences Best Papers)
null
(Ed.)
Identifying people in historical photographs is important for interpreting material culture, correcting the historical record, and creating economic value, but it is also a complex and challenging task. In this paper, we focus on identifying portraits of soldiers who participated in the American Civil War (1861-65). Millions of these portraits survive, but only 10-20% are identified. We created Photo Sleuth, a web-based platform that combines crowdsourced human expertise and automated face recognition to support Civil War portrait identification. Our mixed-methods evaluation of Photo Sleuth one month after its public launch showed that it helped users successfully identify unknown portraits.
Mohanty, Vikram; Thames, David; Luther, Kurt(
, AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018))
We introduce a web-based platform called Civil War Photo
Sleuth for helping users identify unknown soldiers in portraits from the American Civil War era. Our system employs
a novel person identification pipeline by leveraging the complementary strengths of crowdsourced human vision and face
recognition algorithms
Yuan, Liling; Mohanty, Vikram; Luther, Kurt(
, CSCW '21: Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing)
Identifying people in photographs is an important task in many fields, including history, journalism, genealogy, and collecting, but accurate person identification remains challenging. Researchers especially struggle with the “last-mile problem” of historical person identification, where they must make a selection among a small number of highly similar candidates. We present SleuthTalk, a web-based collaboration tool integrated into the public website Civil War Photo Sleuth which addresses the last-mile problem in historical person identification by providing support for shortlisting potential candidates from face recognition results, private collaborative workspaces, and structured feedback.
Identifying people in photographs is a critical task in a wide variety of domains, from national security
[7] to journalism [14] to human rights investigations [1]. The task is also fundamentally complex and
challenging. With the world population at 7.6 billion and growing, the candidate pool is large. Studies
of human face recognition ability show that the average person incorrectly identifies two people as
similar 20–30% of the time, and trained police detectives do not perform significantly better [11].
Computer vision-based face recognition tools have gained considerable ground and are now widely
available commercially, but comparisons to human performance show mixed results at best [2,10,16].
Automated face recognition techniques, while powerful, also have constraints that may be impractical
for many real-world contexts. For example, face recognition systems tend to suffer when the target
image or reference images have poor quality or resolution, as blemishes or discolorations may be
incorrectly recognized as false positives for facial landmarks. Additionally, most face recognition
systems ignore some salient facial features, like scars or other skin characteristics, as well as distinctive
non-facial features, like ear shape or hair or facial hair styles.
This project investigates how we can overcome these limitations to support person identification tasks.
By adjusting confidence thresholds, users of face recognition can generally expect high recall (few false
negatives) at the cost of low precision (many false positives). Therefore, we focus our work on the “last
mile” of person identification, i.e., helping a user find the correct match among a large set of similarlooking candidates suggested by face recognition. Our approach leverages the powerful capabilities of
the human vision system and collaborative sensemaking via crowdsourcing to augment the
complementary strengths of automatic face recognition. The result is a novel technology pipeline
combining collective intelligence and computer vision.
We scope this project to focus on identifying soldiers in photos from the American Civil War era (1861–
1865). An estimated 4,000,000 soldiers fought in the war, and most were photographed at least once,
due to decreasing costs, the increasing robustness of the format, and the critical events separating
friends and family [17]. Over 150 years later, the identities of most of these portraits have been lost,
but as museums and archives increasingly digitize and publish their collections online, the pool of
reference photos and information has never been more accessible. Historians, genealogists, and
collectors work tirelessly to connect names with faces, using largely manual identification methods [3,9].
Identifying people in historical photos is important for preserving material culture [9], correcting the
historical record [13], and recognizing contributions of marginalized groups [4], among other reasons.
Identifying historical photographs of people can generate
significant cultural and economic value, but misidentifications
can cause harms such as falsifying the historical record,
spreading disinformation, and feeding conspiracy theories.
In this paper, we introduce DoubleCheck, a quality assessment
framework based on the concepts of information provenance
and stewardship for verifying historical photo identifications.
We built and evaluated DoubleCheck on Civil War
Photo Sleuth (CWPS), a popular online community dedicated
to identifying photos from the American CivilWar era (1861-
65) using facial recognition and crowdsourcing.
Mohanty, Vikram, Thames, David, Mehta, Sneha, and Luther, Kurt. Photo sleuth: combining human expertise and face recognition to identify historical portraits. Retrieved from https://par.nsf.gov/biblio/10139505. Proceedings of the 24th ACM Conference on Intelligent User Interfaces (IUI ’19) . Web. doi:10.1145/3301275.3302301.
Mohanty, Vikram, Thames, David, Mehta, Sneha, & Luther, Kurt. Photo sleuth: combining human expertise and face recognition to identify historical portraits. Proceedings of the 24th ACM Conference on Intelligent User Interfaces (IUI ’19), (). Retrieved from https://par.nsf.gov/biblio/10139505. https://doi.org/10.1145/3301275.3302301
Mohanty, Vikram, Thames, David, Mehta, Sneha, and Luther, Kurt.
"Photo sleuth: combining human expertise and face recognition to identify historical portraits". Proceedings of the 24th ACM Conference on Intelligent User Interfaces (IUI ’19) (). Country unknown/Code not available. https://doi.org/10.1145/3301275.3302301.https://par.nsf.gov/biblio/10139505.
@article{osti_10139505,
place = {Country unknown/Code not available},
title = {Photo sleuth: combining human expertise and face recognition to identify historical portraits},
url = {https://par.nsf.gov/biblio/10139505},
DOI = {10.1145/3301275.3302301},
abstractNote = {Identifying people in historical photographs is important for preserving material culture, correcting the historical record, and creating economic value, but it is also a complex and challenging task. In this paper, we focus on identifying portraits of soldiers who participated in the American Civil War (1861-65), the first widely-photographed conflict. Many thousands of these portraits survive, but only 10--20% are identified. We created Photo Sleuth, a web-based platform that combines crowdsourced human expertise and automated face recognition to support Civil War portrait identification. Our mixed-methods evaluation of Photo Sleuth one month after its public launch showed that it helped users successfully identify unknown portraits and provided a sustainable model for volunteer contribution. We also discuss implications for crowd-AI interaction and person identification pipelines.},
journal = {Proceedings of the 24th ACM Conference on Intelligent User Interfaces (IUI ’19)},
author = {Mohanty, Vikram and Thames, David and Mehta, Sneha and Luther, Kurt},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.