Photo Sleuth: Combining Collective Intelligence and Computer Vision to Identify Historical Portraits
Identifying people in photographs is a critical task in a wide variety of domains, from national security
[7] to journalism [14] to human rights investigations [1]. The task is also fundamentally complex and
challenging. With the world population at 7.6 billion and growing, the candidate pool is large. Studies
of human face recognition ability show that the average person incorrectly identifies two people as
similar 20–30% of the time, and trained police detectives do not perform significantly better [11].
Computer vision-based face recognition tools have gained considerable ground and are now widely
available commercially, but comparisons to human performance show mixed results at best [2,10,16].
Automated face recognition techniques, while powerful, also have constraints that may be impractical
for many real-world contexts. For example, face recognition systems tend to suffer when the target
image or reference images have poor quality or resolution, as blemishes or discolorations may be
incorrectly recognized as false positives for facial landmarks. Additionally, most face recognition
systems ignore some salient facial features, like scars or other skin characteristics, as well as distinctive
non-facial features, like ear shape or hair or facial hair styles.
This project investigates how we can overcome these limitations to support person identification tasks.
By adjusting confidence thresholds, users of face recognition can generally expect high recall (few false
negatives) at the cost of low precision (many false positives). Therefore, we focus our work on the “last
mile” of person identification, i.e., helping a user find the correct match among a large set of similarlooking candidates suggested by face recognition. Our approach leverages the powerful capabilities of
the human vision system and collaborative sensemaking via crowdsourcing to augment the
complementary strengths of automatic face recognition. The result is a novel technology pipeline
combining collective intelligence and computer vision.
We scope this project to focus on identifying soldiers in photos from the American Civil War era (1861–
1865). An estimated 4,000,000 soldiers fought in the war, and most were photographed at least once,
due to decreasing costs, the increasing robustness of the format, and the critical events separating
friends and family [17]. Over 150 years later, the identities of most of these portraits have been lost,
but as museums and archives increasingly digitize and publish their collections online, the pool of
reference photos and information has never been more accessible. Historians, genealogists, and
collectors work tirelessly to connect names with faces, using largely manual identification methods [3,9].
Identifying people in historical photos is important for preserving material culture [9], correcting the
historical record [13], and recognizing contributions of marginalized groups [4], among other reasons.
more »
« less
