This dataset contains machine learning and volunteer classifications from the Gravity Spy project. It includes glitches from observing runs O1, O2, O3a and O3b that received at least one classification from a registered volunteer in the project. It also indicates glitches that are nominally retired from the project using our default set of retirement parameters, which are described below. See more details in the Gravity Spy Methods paper.
When a particular subject in a citizen science project (in this case, glitches from the LIGO datastream) is deemed to be classified sufficiently it is "retired" from the project. For the Gravity Spy project, retirement depends on a combination of both volunteer and machine learning classifications, and a number of parameterizations affect how quickly glitches get retired. For this dataset, we use a default set of retirement parameters, the most important of which are:
- A glitches must be classified by at least 2 registered volunteers
- Based on both the initial machine learning classification and volunteer classifications, the glitch has more than a 90% probability of residing in a particular class
- Each volunteer classification (weighted by that volunteer's confusion matrix) contains a weight equal to the initial machine learning score when determining the final probability
The choice of these and other parameterization will affect the accuracy of the retired dataset as well as the number of glitches that are retired, and will be explored in detail in an upcoming publication (Zevin et al. in prep).
The dataset can be read in using e.g. Pandas:
```
import pandas as pd
dataset = pd.read_hdf('retired_fulldata_min2_max50_ret0p9.hdf5', key='image_db')
```
Each row in the dataframe contains information about a particular glitch in the Gravity Spy dataset.
Description of series in dataframe
- ['1080Lines', '1400Ripples', 'Air_Compressor', 'Blip', 'Chirp', 'Extremely_Loud', 'Helix', 'Koi_Fish', 'Light_Modulation', 'Low_Frequency_Burst', 'Low_Frequency_Lines', 'No_Glitch', 'None_of_the_Above', 'Paired_Doves', 'Power_Line', 'Repeating_Blips', 'Scattered_Light', 'Scratchy', 'Tomte', 'Violin_Mode', 'Wandering_Line', 'Whistle']
- Machine learning scores for each glitch class in the trained model, which for a particular glitch will sum to unity
- ['ml_confidence', 'ml_label']
- Highest machine learning confidence score across all classes for a particular glitch, and the class associated with this score
- ['gravityspy_id', 'id']
- Unique identified for each glitch on the Zooniverse platform ('gravityspy_id') and in the Gravity Spy project ('id'), which can be used to link a particular glitch to the full Gravity Spy dataset (which contains GPS times among many other descriptors)
- ['retired']
- Marks whether the glitch is retired using our default set of retirement parameters (1=retired, 0=not retired)
- ['Nclassifications']
- The total number of classifications performed by registered volunteers on this glitch
- ['final_score', 'final_label']
- The final score (weighted combination of machine learning and volunteer classifications) and the most probable type of glitch
- ['tracks']
- Array of classification weights that were added to each glitch category due to each volunteer's classification
```
For machine learning classifications on all glitches in O1, O2, O3a, and O3b, please see Gravity Spy Machine Learning Classifications on Zenodo
For the most recently uploaded training set used in Gravity Spy machine learning algorithms, please see Gravity Spy Training Set on Zenodo.
For detailed information on the training set used for the original Gravity Spy machine learning paper, please see Machine learning for Gravity Spy: Glitch classification and dataset on Zenodo.
