Human gaze behavior prediction is important for behavioral vision and for computer vision applications. Most models mainly focus on predicting free-viewing behavior using saliency maps, but do not generalize to goal-directed behavior, such as when a person searches for a visual target object. We propose the first inverse reinforcement learning (IRL) model to learn the internal reward function and policy used by humans during visual search. We modeled the viewer’s internal belief states as dynamic contextual belief maps of object locations. These maps were learned and then used to predict behavioral scanpaths for multiple target categories. To train and evaluate our IRL model we created COCO-Search18, which is now the largest dataset of highquality search fixations in existence. COCO-Search18 has 10 participants searching for each of 18 target-object categories in 6202 images, making about 300,000 goal-directed fixations. When trained and evaluated on COCO-Search18, the IRL model outperformed baseline models in predicting search fixation scanpaths, both in terms of similarity to human search behavior and search efficiency. Finally, reward maps recovered by the IRL model reveal distinctive targetdependent patterns of object prioritization, which we interpret as a learned object context.
more »
« less
COCO-Search18 fixation dataset for predicting goal-directed attention control
Abstract Attention control is a basic behavioral process that has been studied for decades. The currently best models of attention control are deep networks trained on free-viewing behavior to predict bottom-up attention control – saliency. We introduce COCO-Search18, the first dataset of laboratory-qualitygoal-directed behaviorlarge enough to train deep-network models. We collected eye-movement behavior from 10 people searching for each of 18 target-object categories in 6202 natural-scene images, yielding$$\sim$$ 300,000 search fixations. We thoroughly characterize COCO-Search18, and benchmark it using three machine-learning methods: a ResNet50 object detector, a ResNet50 trained on fixation-density maps, and an inverse-reinforcement-learning model trained on behavioral search scanpaths. Models were also trained/tested on images transformed to approximate a foveated retina, a fundamental biological constraint. These models, each having a different reliance on behavioral training, collectively comprise the new state-of-the-art in predicting goal-directed search fixations. Our expectation is that future work using COCO-Search18 will far surpass these initial efforts, finding applications in domains ranging from human-computer interactive systems that can anticipate a person’s intent and render assistance to the potentially early identification of attention-related clinical disorders (ADHD, PTSD, phobia) based on deviation from neurotypical fixation behavior.
more »
« less
- PAR ID:
- 10223276
- Publisher / Repository:
- Nature Publishing Group
- Date Published:
- Journal Name:
- Scientific Reports
- Volume:
- 11
- Issue:
- 1
- ISSN:
- 2045-2322
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract This paper presents a search for dark matter,$$\chi $$ , using events with a single top quark and an energeticWboson. The analysis is based on proton–proton collision data collected with the ATLAS experiment at$$\sqrt{s}=$$ 13 TeV during LHC Run 2 (2015–2018), corresponding to an integrated luminosity of 139 fb$$^{-1}$$ . The search considers final states with zero or one charged lepton (electron or muon), at least oneb-jet and large missing transverse momentum. In addition, a result from a previous search considering two-charged-lepton final states is included in the interpretation of the results. The data are found to be in good agreement with the Standard Model predictions and the results are interpreted in terms of 95% confidence-level exclusion limits in the context of a class of dark matter models involving an extended two-Higgs-doublet sector together with a pseudoscalar mediator particle. The search is particularly sensitive to on-shell production of the charged Higgs boson state,$$H^{\pm }$$ , arising from the two-Higgs-doublet mixing, and its semi-invisible decays via the mediator particle,a:$$H^{\pm } \rightarrow W^\pm a (\rightarrow \chi \chi )$$ . Signal models with$$H^{\pm }$$ masses up to 1.5 TeV andamasses up to 350 GeV are excluded assuming a$$\tan \beta $$ value of 1. For masses ofaof 150 (250) GeV,$$\tan \beta $$ values up to 2 are excluded for$$H^{\pm }$$ masses between 200 (400) GeV and 1.5 TeV. Signals with$$\tan \beta $$ values between 20 and 30 are excluded for$$H^{\pm }$$ masses between 500 and 800 GeV.more » « less
-
Abstract This paper reports a search for Higgs boson pair (hh) production in association with a vector boson ($$W\; {\text {o}r}\; Z$$ ) using 139 fb$$^{-1}$$ of proton–proton collision data at$$\sqrt{s}=13\,\text {TeV}$$ recorded with the ATLAS detector at the Large Hadron Collider. The search is performed in final states in which the vector boson decays leptonically ($$W\rightarrow \ell \nu ,\, Z\rightarrow \ell \ell ,\nu \nu $$ with$$\ell =e, \mu $$ ) and the Higgs bosons each decay into a pair ofb-quarks. It targetsVhhsignals from both non-resonanthhproduction, present in the Standard Model (SM), and resonanthhproduction, as predicted in some SM extensions. A 95% confidence-level upper limit of 183 (87) times the SM cross-section is observed (expected) for non-resonantVhhproduction when assuming the kinematics are as expected in the SM. Constraints are also placed on Higgs boson coupling modifiers. For the resonant search, upper limits on the production cross-sections are derived for two specific models: one is the production of a vector boson along with a neutral heavy scalar resonanceH, in the mass range 260–1000 GeV, that decays intohh, and the other is the production of a heavier neutral pseudoscalar resonanceAthat decays into aZboson andHboson, where theAboson mass is 360–800 GeV and theHboson mass is 260–400 GeV. Constraints are also derived in the parameter space of two-Higgs-doublet models.more » « less
-
Abstract A search for supersymmetry involving the pair production of gluinos decaying via off-shell third-generation squarks into the lightest neutralino$$(\tilde{\chi }^0_1)$$ is reported. It exploits LHC proton–proton collision data at a centre-of-mass energy$$\sqrt{s} = 13$$ TeV with an integrated luminosity of 139 fb$$^{-1}$$ collected with the ATLAS detector from 2015 to 2018. The search uses events containing large missing transverse momentum, up to one electron or muon, and several energetic jets, at least three of which must be identified as containingb-hadrons. Both a simple kinematic event selection and an event selection based upon a deep neural-network are used. No significant excess above the predicted background is found. In simplified models involving the pair production of gluinos that decay via off-shell top (bottom) squarks, gluino masses less than 2.44 TeV (2.35 TeV) are excluded at 95% CL for a massless$$\tilde{\chi }^0_1.$$ Limits are also set on the gluino mass in models with variable branching ratios for gluino decays to$$b\bar{b}\tilde{\chi }^0_1,$$ $$t\bar{t}\tilde{\chi }^0_1$$ and$$t\bar{b}\tilde{\chi }^-_1/\bar{t}b\tilde{\chi }^+_1.$$more » « less
-
Abstract A search for pair-produced vector-like quarks using events with exactly one lepton (eor$$\mu $$ ), at least four jets including at least oneb-tagged jet, and large missing transverse momentum is presented. Data from proton–proton collisions at a centre-of-mass energy of$$\sqrt{s}=$$ 13 $$\text {TeV}$$ , recorded by the ATLAS detector at the LHC from 2015 to 2018 and corresponding to an integrated luminosity of 139 fb$$^{-1}$$ , are analysed. Vector-like partnersTandBof the top and bottom quarks are considered, as is a vector-likeXwith charge$$+5/3$$ , assuming their decay into aW,Z, or Higgs boson and a third-generation quark. No significant deviations from the Standard Model expectation are observed. Upper limits on the production cross-section ofTandBquark pairs as a function of their mass are derived for various decay branching ratio scenarios. The strongest lower limits on the masses are 1.59 $$\text {TeV}$$ assuming mass-degenerate vector-like quarks and branching ratios corresponding to the weak-isospin doublet model, and 1.47 $$\text {TeV}$$ (1.46 $$\text {TeV}$$ ) for exclusive$$T \rightarrow Zt$$ ($$B/X \rightarrow Wt$$ ) decays. In addition, lower limits on theTandBquark masses are derived for all possible branching ratios.more » « less