skip to main content

Title: The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics
A new paradigm for data-driven, model-agnostic new physics searches at colliders is emerging, and aims to leverage recent breakthroughs in anomaly detection and machine learning. In order to develop and benchmark new anomaly detection methods within this framework, it is essential to have standard datasets. To this end, we have created the LHC Olympics 2020, a community challenge accompanied by a set of simulated collider events. Participants in these Olympics have developed their methods using an R&D dataset and then tested them on black boxes: datasets with an unknown anomaly (or not). This paper will review the LHC Olympics 2020 challenge, including an overview of the competition, a description of methods deployed in the competition, lessons learned from the experience, and implications for data analyses with future datasets as well as future colliders.
; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; more » ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; « less
Kasieczka, Gregor; Nachman, Benjamin; Shih, David
Award ID(s):
Publication Date:
Journal Name:
Sponsoring Org:
National Science Foundation
More Like this
  1. We describe the outcome of a data challenge conducted as part of the Dark Machines ( initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenged aims to detect signals of new physics at the Large Hadron Collider (LHC) using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of >1 billion simulated LHC events corresponding to 10\, fb^{-1} 10 f b − 1 of proton-proton collisions at a center-of-mass energy of 13 TeV.more »We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC, and provide our benchmark dataset for future studies at Code to reproduce the analysis is provided at« less
  2. Abstract With the establishment and maturation of the experimental programs searching for new physics with sizeable couplings at the LHC, there is an increasing interest in the broader particle and astrophysics community for exploring the physics of light and feebly-interacting particles as a paradigm complementary to a New Physics sector at the TeV scale and beyond. FIPs 2020 has been the first workshop fully dedicated to the physics of feebly-interacting particles and was held virtually from 31 August to 4 September 2020. The workshop has gathered together experts from collider, beam dump, fixed target experiments, as well as from astrophysics,more »axions/ALPs searches, current/future neutrino experiments, and dark matter direct detection communities to discuss progress in experimental searches and underlying theory models for FIPs physics, and to enhance the cross-fertilisation across different fields. FIPs 2020 has been complemented by the topical workshop “Physics Beyond Colliders meets theory”, held at CERN from 7 June to 9 June 2020. This document presents the summary of the talks presented at the workshops and the outcome of the subsequent discussions held immediately after. It aims to provide a clear picture of this blooming field and proposes a few recommendations for the next round of experimental results.« less
  3. Neutrinos from a particle collider have never been directly detected. FASER𝜈 at the Large Hadron Collider (LHC) is designed to detect such neutrinos for the first time and study their cross sections at TeV energies—at present, no such measurements are available at such high energies. In 2018, during LHC Run 2, we installed a pilot detector 480-m downstream of the ATLAS interaction point. In this pilot run, proton–proton collision data of 12.2 fb−1 at a center-of-mass energy of 13 TeV were collected. We observed the first candidate vertices, which were consistent with neutrino interactions. A 2.7𝜎 excess of neutrino-like signalmore »above the background was measured. This milestone opens a new avenue for studying neutrinos at the existing and future high-energy colliders. During LHC Run 3, which will commence in 2022, we will deploy an emulsion detector with a target mass of 1.1 tons, coupled with the FASER magnetic spectrometer. This will yield ∼2,000 𝜈𝑒, ∼6,000 𝜈𝜇, and ∼40 𝜈𝜏 interactions in the detector. Herein, we present the status and plan of FASER𝜈 and report neutrino detection in the 2018 data.« less
  4. Abstract A precise knowledge of the quark and gluon structure of the proton, encoded by the parton distribution functions (PDFs), is of paramount importance for the interpretation of high-energy processes at present and future lepton–hadron and hadron–hadron colliders. Motivated by recent progress in the PDF determinations carried out by the CT, MSHT, and NNPDF groups, we present an updated combination of global PDF fits: PDF4LHC21. It is based on the Monte Carlo combination of the CT18, MSHT20, and NNPDF3.1 sets followed by either its Hessian reduction or its replica compression. Extensive benchmark studies are carried out in order to disentanglemore »the origin of the differences between the three global PDF sets. In particular, dedicated fits based on almost identical theory settings and input datasets are performed by the three groups, highlighting the role played by the respective fitting methodologies. We compare the new PDF4LHC21 combination with its predecessor, PDF4LHC15, demonstrating their good overall consistency and a modest reduction of PDF uncertainties for key LHC processes such as electroweak gauge boson production and Higgs boson production in gluon fusion. We study the phenomenological implications of PDF4LHC21 for a representative selection of inclusive, fiducial, and differential cross sections at the LHC. The PDF4LHC21 combination is made available via the LHAPDF library and provides a robust, user-friendly, and efficient method to estimate the PDF uncertainties associated to theoretical calculations for the upcoming Run III of the LHC and beyond.« less
  5. Obeid, Iyad ; Selesnick, Ivan ; Picone, Joseph (Ed.)
    The Temple University Hospital Seizure Detection Corpus (TUSZ) [1] has been in distribution since April 2017. It is a subset of the TUH EEG Corpus (TUEG) [2] and the most frequently requested corpus from our 3,000+ subscribers. It was recently featured as the challenge task in the Neureka 2020 Epilepsy Challenge [3]. A summary of the development of the corpus is shown below in Table 1. The TUSZ Corpus is a fully annotated corpus, which means every seizure event that occurs within its files has been annotated. The data is selected from TUEG using a screening process that identifies filesmore »most likely to contain seizures [1]. Approximately 7% of the TUEG data contains a seizure event, so it is important we triage TUEG for high yield data. One hour of EEG data requires approximately one hour of human labor to complete annotation using the pipeline described below, so it is important from a financial standpoint that we accurately triage data. A summary of the labels being used to annotate the data is shown in Table 2. Certain standards are put into place to optimize the annotation process while not sacrificing consistency. Due to the nature of EEG recordings, some records start off with a segment of calibration. This portion of the EEG is instantly recognizable and transitions from what resembles lead artifact to a flat line on all the channels. For the sake of seizure annotation, the calibration is ignored, and no time is wasted on it. During the identification of seizure events, a hard “3 second rule” is used to determine whether two events should be combined into a single larger event. This greatly reduces the time that it takes to annotate a file with multiple events occurring in succession. In addition to the required minimum 3 second gap between seizures, part of our standard dictates that no seizure less than 3 seconds be annotated. Although there is no universally accepted definition for how long a seizure must be, we find that it is difficult to discern with confidence between burst suppression or other morphologically similar impressions when the event is only a couple seconds long. This is due to several reasons, the most notable being the lack of evolution which is oftentimes crucial for the determination of a seizure. After the EEG files have been triaged, a team of annotators at NEDC is provided with the files to begin data annotation. An example of an annotation is shown in Figure 1. A summary of the workflow for our annotation process is shown in Figure 2. Several passes are performed over the data to ensure the annotations are accurate. Each file undergoes three passes to ensure that no seizures were missed or misidentified. The first pass of TUSZ involves identifying which files contain seizures and annotating them using our annotation tool. The time it takes to fully annotate a file can vary drastically depending on the specific characteristics of each file; however, on average a file containing multiple seizures takes 7 minutes to fully annotate. This includes the time that it takes to read the patient report as well as traverse through the entire file. Once an event has been identified, the start and stop time for the seizure is stored in our annotation tool. This is done on a channel by channel basis resulting in an accurate representation of the seizure spreading across different parts of the brain. Files that do not contain any seizures take approximately 3 minutes to complete. Even though there is no annotation being made, the file is still carefully examined to make sure that nothing was overlooked. In addition to solely scrolling through a file from start to finish, a file is often examined through different lenses. Depending on the situation, low pass filters are used, as well as increasing the amplitude of certain channels. These techniques are never used in isolation and are meant to further increase our confidence that nothing was missed. Once each file in a given set has been looked at once, the annotators start the review process. The reviewer checks a file and comments any changes that they recommend. This takes about 3 minutes per seizure containing file, which is significantly less time than the first pass. After each file has been commented on, the third pass commences. This step takes about 5 minutes per seizure file and requires the reviewer to accept or reject the changes that the second reviewer suggested. Since tangible changes are made to the annotation using the annotation tool, this step takes a bit longer than the previous one. Assuming 18% of the files contain seizures, a set of 1,000 files takes roughly 127 work hours to annotate. Before an annotator contributes to the data interpretation pipeline, they are trained for several weeks on previous datasets. A new annotator is able to be trained using data that resembles what they would see under normal circumstances. An additional benefit of using released data to train is that it serves as a means of constantly checking our work. If a trainee stumbles across an event that was not previously annotated, it is promptly added, and the data release is updated. It takes about three months to train an annotator to a point where their annotations can be trusted. Even though we carefully screen potential annotators during the hiring process, only about 25% of the annotators we hire survive more than one year doing this work. To ensure that the annotators are consistent in their annotations, the team conducts an interrater agreement evaluation periodically to ensure that there is a consensus within the team. The annotation standards are discussed in Ochal et al. [4]. An extended discussion of interrater agreement can be found in Shah et al. [5]. The most recent release of TUSZ, v1.5.2, represents our efforts to review the quality of the annotations for two upcoming challenges we hosted: an internal deep learning challenge at IBM [6] and the Neureka 2020 Epilepsy Challenge [3]. One of the biggest changes that was made to the annotations was the imposition of a stricter standard for determining the start and stop time of a seizure. Although evolution is still included in the annotations, the start times were altered to start when the spike-wave pattern becomes distinct as opposed to merely when the signal starts to shift from background. This cuts down on background that was mislabeled as a seizure. For seizure end times, all post ictal slowing that was included was removed. The recent release of v1.5.2 did not include any additional data files. Two EEG files had been added because, originally, they were corrupted in v1.5.1 but were able to be retrieved and added for the latest release. The progression from v1.5.0 to v1.5.1 and later to v1.5.2, included the re-annotation of all of the EEG files in order to develop a confident dataset regarding seizure identification. Starting with v1.4.0, we have also developed a blind evaluation set that is withheld for use in competitions. The annotation team is currently working on the next release for TUSZ, v1.6.0, which is expected to occur in August 2020. It will include new data from 2016 to mid-2019. This release will contain 2,296 files from 2016 as well as several thousand files representing the remaining data through mid-2019. In addition to files that were obtained with our standard triaging process, a part of this release consists of EEG files that do not have associated patient reports. Since actual seizure events are in short supply, we are mining a large chunk of data for which we have EEG recordings but no reports. Some of this data contains interesting seizure events collected during long-term EEG sessions or data collected from patients with a history of frequent seizures. It is being mined to increase the number of files in the corpus that have at least one seizure event. We expect v1.6.0 to be released before IEEE SPMB 2020. The TUAR Corpus is an open-source database that is currently available for use by any registered member of our consortium. To register and receive access, please follow the instructions provided at this web page: The data is located here:« less