An official website of the United States government
Abstract “Paintings fade like flowers”: van Gogh’s prediction on the impact of age on paintings came true for most of his paintings. We have studied the consequences of this aging on the Sunflowers in a vase with a yellow background series, namely its original, F454, currently in London, and two replicates, F457, in Tokyo, and F458, in Amsterdam, which van Gogh painted using the original as a model. The background and flower renditions in those paintings have faded and turned brown, making them less vibrant that van Gogh had most likely intended. We have attempted to restore van Gogh’s intent using a computational approach based on data science. After identifications of regions of interest (ROI) within the three paintings F454, F457, and F458 that capture the flowers, stems of the flowers, and background, respectively, we studied the geometry of the color space (in RGB representation) occupied by those ROIs. By comparing those color spaces with those occupied by similar ROIs in photographs of real sunflowers, we identified shifts in all three color coordinates, R, G, and B, with the positive shift in the blue coordinate being the more salient. We have proposed two algorithms, PCR-1 and PCR-2, for correcting that shift in blue and generate representations of the paintings that aim to restore their original conditions. The reduction of the blue component in the yellow hues has lead to more vibrant and less brownish digital rendition of the three Sunflowers in a vase with a yellow background .
more »
« less
A Machine Learning Paradigm for Studying Pictorial Realism: How Accurate are Constable's Clouds?
The British landscape painter John Constable is considered foundational for the Realist movement in 19th-century European painting. Constable’s painted skies, in particular, were seen as remarkably accurate by his contemporaries, an impression shared by many viewers today. Yet, assessing the accuracy of realist paintings like Constable’s is subjective or intuitive, even for professional art historians, making it difficult to say with certainty what set Constable’s skies apart from those of his contemporaries. Our goal is to contribute to a more objective understanding of Constable’s realism. We propose a new machine-learning-based paradigm for studying pictorial realism in an explainable way. Our framework assesses realism by measuring the similarity between clouds painted by artists noted for their skies, like Constable, and photographs of clouds. The experimental results of cloud classification show that Constable approximates more consistently than his contemporaries the formal features of actual clouds in his paintings. The study, as a novel interdisciplinary approach that combines computer vision and machine learning, meteorology, and art history, is a springboard for broader and deeper analyses of pictorial realism.
more »
« less
- Award ID(s):
- 2234195
- PAR ID:
- 10493018
- Publisher / Repository:
- IEEE
- Date Published:
- Journal Name:
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Volume:
- 46
- Issue:
- 1
- ISSN:
- 0162-8828
- Page Range / eLocation ID:
- 33 to 42
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract As we contemplate the future of forest landscapes under changing climate conditions and land‐use demands, there is increasing value in studying historic forest conditions and how these landscapes have changed following past disturbances. Historic landscape paintings are a potential source of data on preindustrial forests with highly detailed, full‐color depictions of overstory and understory environments. They display key details about forest community composition, microhabitat features, and structural complexity from a time well before the advent of color photography. Despite these paintings' potential, their scientific applications have been impeded by questions of validity. How truly accurate are the images portrayed in these paintings? How much of an image is an artist's manipulation of a scene to best illustrate an allegory or romanticized view of nature? Following an established assessment model from historical ecology for evaluating resource validity, we demonstrate how scholarship on art history can be integrated with ecological understanding of forest landscapes to follow this model and address these questions of image veracity in 19th century American art. Further, to illustrate the potential use of these historic images in ecological studies, we present in a case study assessing microhabitat features of 10 different paintings. While this paper explores 19th century landscape art broadly, we focus our art historical review in particular on Asher Durand, a prolific and influential artist associated with the so‐called “Hudson River School” in the mid‐1800s. Durand left clear records about his perspectives on accurately depicting nature, and from a review of images and writings of Durand, we find support for the potential use of many of his paintings and sketches in historic forest ecology research. However, we also identify important caveats regarding potential ecological interpretations from these images. More broadly, because 19th century landscape paintings are not always directly transcriptive, and because regional art cultures differed in the 1800s, we cannot within this paper speak about landscape image veracity across all 19th century landscape art. However, in following established methods in historical ecology and integrating tools from art history research, we show that one can identify accurate historic landscape paintings for application in scientific studies.more » « less
-
Al-Kadi, Omar Sultan (Ed.)In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists’ eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl . We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.more » « less
-
null (Ed.)As inborn characteristics, humans possess the ability to judge visual aesthetics, feel the emotions from the environment, and comprehend others’ emotional expressions. Many exciting applications become possible if robots or computers can be empowered with similar capabilities. Modeling aesthetics, evoked emotions, and emotional expressions automatically in unconstrained situations, however, is daunting due to the lack of a full understanding of the relationship between low-level visual content and high-level aesthetics or emotional expressions. With the growing availability of data, it is possible to tackle these problems using machine learning and statistical modeling approaches. In the talk, I provide an overview of our research in the last two decades on data-driven analyses of visual artworks and digital visual content for modeling aesthetics and emotions. First, I discuss our analyses of styles in visual artworks. Art historians have long observed the highly characteristic brushstroke styles of Vincent van Gogh and have relied on discerning these styles for authenticating and dating his works. In our work, we compared van Gogh with his contemporaries by statistically analyzing a massive set of automatically extracted brushstrokes. A novel extraction method is developed by exploiting an integration of edge detection and clustering-based segmentation. Evidence substantiates that van Gogh’s brushstrokes are strongly rhythmic. Next, I describe an effort to model the aesthetic and emotional characteristics in visual contents such as photographs. By taking a data-driven approach, using the Internet as the data source, we show that computers can be trained to recognize various characteristics that are highly relevant to aesthetics and emotions. Future computer systems equipped with such capabilities are expected to help millions of users in unimagined ways. Finally, I highlight our research on automated recognition of bodily expression of emotion. We propose a scalable and reliable crowdsourcing approach for collecting in-the-wild perceived emotion data for computers to learn to recognize the body language of humans. Comprehensive statistical analysis revealed many interesting insights from the dataset. A system to model the emotional expressions based on bodily movements, named ARBEE (Automated Recognition of Bodily Expression of Emotion), has also been developed and evaluated.more » « less
-
The medial temporal lobe (MTL) is traditionally considered to be a system that is specialized for long-term memory. Recent work has challenged this notion by demonstrating that this region can contribute to many domains of cognition beyond long-term memory, including perception and attention. One potential reason why the MTL (and hippocampus specifically) contributes broadly to cognition is that it contains relational representations—representations of multidimensional features of experience and their unique relationship to one another—that are useful in many different cognitive domains. Here, we explore the hypothesis that the hippocampus/MTL plays a critical role in attention and perception via relational representations. We compared human participants with MTL damage to healthy age- and education-matched individuals on attention tasks that varied in relational processing demands. On each trial, participants viewed two images (rooms with paintings). On “similar room” trials, they judged whether the rooms had the same spatial layout from a different perspective. On “similar art” trials, they judged whether the paintings could have been painted by the same artist. On “identical” trials, participants simply had to detect identical paintings or rooms. MTL lesion patients were significantly and selectively impaired on the similar room task. This work provides further evidence that the hippocampus/MTL plays a ubiquitous role in cognition by virtue of its relational and spatial representations and highlights its important contributions to rapid perceptual processes that benefit from attention.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1109/TPAMI.2023.3324743
